yacy_search_server/Heroku.md

Yacy on Heroku

YaCy can be deployed on Heroku platform in various ways. There are currently limitations which make this deployment option better suited for demonstration or testing purposes.

Limitations

### Ephemeral file system

Each Heroku container instance (aka "dyno") file system is ephemeral : all files are discarded each time the "dyno" is restarted (at least automatically once a day by the platform).

This is not a problem for applications storing their persistent data in an external cloud datastore. But YaCy is currently designed to use the local file system. So beware that when running YaCy on Heroku, all your settings and indexed data will be lost once a day.

Build version

YaCy main git history is too large to be pushed on Heroku (more than 300MB Heroku "slug" limit), but it is normally used to produce the revision number suffix added to main version. So when building on Heroku platform, YaCy version in /Status.html will always appear with a default revision number (for example 1.91/9000).

Deploy with the button

• Click on the 'Deploy to Heroku' button on the main YaCy README.md.
• Log in with your Heroku account or create one.
• A preconfigured deploy page is proposed (configuration comes from the app.json file).
• Enter the name of your application (don't let Heroku choose a default one).
• Edit the configuration variable YACY_PUBLIC_URL : fill it with a URL like your_app_name.herokuapp.com, with your_app_name replaced with the name you chose. If you ignore this step, YaCy will run, but in junior mode : it will not be able to be reached by other peers and will not contribute to the global indexing.
• Edit the configuration variable YACY_INIT_ADMIN_PASSWORD : fill it with your custom admin password encoded with YaCy (you can get this encoded value by running a local YaCy peer, setting your custom admin password in /ConfigAccounts_p.html, and retrieving it in DATA/SETTINGS/yacy.conf at key adminAccountBase64MD5).
• Click on the deploy button.
• Heroku now build YaCy from sources.
• If everything went fine, you can open YaCy search page with the View button

## Deploy from command line

Here are some brief instructions to deploy YaCy on Heroku from command line. More detailed explanations can be found on related Heroku documentation.

• Install the Heroku Toolbelt
• Get YaCy sources from git as a zip archive, or be sure to remove the .git directory (if you cloned from git, the .git directory will be far too large and later you will not be able to push sources to Heroku).
• Optional steps (deploy locally to check everything is fine) :
  • build with maven : mvn clean dependency:list install -DskipTests=true -f libbuild/pom.xml
  • run locally : heroku local
  • check everything works fine at http://localhost:8090
  • stop the local YaCy
• Log in on heroku : heroku login
• Create an app on heroku : heroku create [your_app_name]
• Initialize a git repository : git init
• Add remote heroku git repository for this deployment : heroku git:remote -a your_app_name
• Set the MAVEN_CUSTOM_OPTS config var : heroku config:set MAVEN_CUSTOM_OPTS="-f libbuild/pom.xml -DskipTests=true"
• Set the YACY_INIT_ADMIN_PASSWORD config var : heroku config:set YACY_INIT_ADMIN_PASSWORD="MD5:[your_encoded_password]"
• Set the YACY_PUBLIC_URL config var : heroku config:set YACY_PUBLIC_URL="[your_app_name].herokuapp.com"
• Add files to git index : git add .
• Commit : git commit
• Push to heroku : git push heroku master
• Open app on your browser : heroku open -a your_app_name

Deploy with GitHub account

• Log in on Heroku
• Click the 'New > Create new app' button
• Eventually choose your app name and your region, then click 'Create App'
• Go to the 'Settings' tab
• Click the 'Reveal Config Vars' button, and add vars MAVEN_CUSTOM_OPTS, YACY_INIT_ADMIN_PASSWORD, YACY_PUBLIC_URL with values filled as described in the previous paragraph
• Go to the 'Deploy' tab
• Choose 'GitHub' deployment method
• Enter your GitHub account information
• Select your YaCy repository clone and 'Connect'
• Choose either automatic or manual deploy, and click the deploy button
• If everything went fine, you can open YaCy search page with the 'View' button

Technical details

Heroku specific configuration files :

• app.json : used when deploying with the button
• .env : set up environment variables for local heroku run
• Procfile : contain main process description used to launch YaCy

Custom maven options

With any of the deployment methods described, setting the option -f libbuild/pom.xml -DskipTests=true in the MAVEN_CUSTOM_OPTS environment variable is the minimum required for a successful build and deploy. If not set, build will fail because missing dependent submodules from libbuild directory.

What's more, the only way for other YaCy peers to reach a peer running on Heroku is to use the "dyno" public URL (in the form of your_app_name.herokuapp.com). This is why the configuration variable YACY_PUBLIC_URL as to be set, or else your YaCy peer will run in "junior" mode. This variable is used in the Procfile to customize the staticIP initial property in the yacy.init file at launch.

HTTP local port

On heroku platform, you can not choose your application binding port. It is set by Heroku in the PORT environment variable, and at startup, the application has to bind to this port within 90 seconds. So YaCy has to bind to this port each time the "dyno" is started, and thus can not rely on its normal configuration file. This is done with the JVM system property net.yacy.server.localPort set in Procfile.

Administrator password

If you wish to use YaCy administration features, you have to set an admin password, and authenticate with it. To generate the encoded password hash set in the config var YACY_INIT_ADMIN_PASSWORD, you can proceed as follows:

• either run a local YaCy peer, set your custom admin password in /ConfigAccounts_p.html, and retrieve it in DATA/SETTINGS/yacy.conf at key adminAccountBase64MD5
• OR run this command in YaCy source directory :
• java -classpath target/classes:lib/* net.yacy.cora.order.Digest -strfhex "admin:The YaCy access is limited to administrators. If you don't know the password, you can change it using <yacy-home>/bin/passwd.sh <new-password>:[your_password]"