Merge pull request #61 from luccioman/heroku_experiments

Deploy YaCy on Heroku
pull/71/head
Orbiter 9 years ago committed by GitHub
commit 503312ca43

@ -0,0 +1,2 @@
MAVEN_CUSTOM_OPTS=-f libbuild/pom.xml -DskipTests=true
PORT=8090

@ -0,0 +1,91 @@
# Yacy on Heroku
YaCy can be deployed on Heroku platform in various ways. There are currently limitations which make this deployment option better suited for demonstration or testing purposes.
## Limitations
### Ephemeral file system
Each Heroku container instance (aka ["dyno"](https://devcenter.heroku.com/articles/dynos#dynos)) file system is [ephemeral](https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem) : all files are discarded each time the "dyno" is restarted (at least automatically once a day by the platform).
This is not a problem for applications storing their persistent data in an external cloud datastore. But YaCy is currently designed to use the local file system. So beware that when running YaCy on Heroku, all your settings and indexed data will be lost once a day.
### Build version
YaCy main git history is too large to be pushed on Heroku (more than 300MB Heroku "slug" limit), but it is normally used to produce the revision number suffix added to main version. So when building on Heroku platform, YaCy version in /Status.html will always appears with a default revision number (for example 1.91/9000).
## Deploy with the button
- Click on the 'Deploy to Heroku' button on the main YaCy [README.md](README.md).
- Log in with your Heroku account or create one.
- A preconfigured deploy page is proposed (configuration comes from the [app.json](app.json) file).
- Enter the name of your application (don't let Heroku choose a default one).
- Edit the configuration variable `YACY_PUBLIC_URL` : fill it with an URL like `your_app_name.herokuapp.com`, with `your_app_name` replaced with the name you choosed.
If you ignore this step, YaCy will run, but in junior mode : it will not be able to be reached by other peers and will not contribute to the global indexing.
- Edit the configuration variable `YACY_INIT_ADMIN_PASSWORD` : fill it with your custom admin password encoded with YaCy (you can get this encoded value by running a local YaCy peer, setting your custom admin password in /ConfigAccounts_p.html, and retrieving it in DATA/SETTINGS/yacy.conf at key `adminAccountBase64MD5`).
- Click on the deploy button.
- Heroku now build YaCy from sources.
- If everything went fine, you can open YaCy search page with the `View` button
## Deploy from command line
Here are some brief instructions to deploy YaCy on Heroku from command line. More detailed explanations can be found on related [Heroku documentation](https://devcenter.heroku.com/articles/getting-started-with-java#introduction).
- Install the Heroku Toolbelt
- Get YaCy sources from git as a zip archive, or be sure to remove the .git directory (if you cloned from git, the .git directory will be far too large and later you will not be able to push sources to Heroku).
- Optional steps (deploy locally to check everything is fine) :
- build with maven : `mvn clean dependency:list install -DskipTests=true -f libbuild/pom.xml`
- run locally : `heroku local`
- check everything works fine at http://localhost:8090
- stop the local YaCy
- Log in on heroku : `heroku login`
- Create an app on heroku : `heroku create [your_app_name]`
- Initialize a git repository : `git init`
- Add remote heroku git repository for this deployment : `heroku git:remote -a your_app_name`
- Set the `MAVEN_CUSTOM_OPTS` config var : `heroku config:set MAVEN_CUSTOM_OPTS="-f libbuild/pom.xml -DskipTests=true"`
- Set the `YACY_INIT_ADMIN_PASSWORD` config var : `heroku config:set YACY_INIT_ADMIN_PASSWORD="MD5:[your_encoded_password]"`
- Set the `YACY_PUBLIC_URL` config var : `heroku config:set YACY_PUBLIC_URL="[your_app_name].herokuapp.com"`
- Add files to git index : `git add .`
- Commit : `git commit`
- Push to heroku : `git push heroku master`
- Open app on your browser : `heroku open -a your_app_name`
## Deploy with GitHub account
- Log in on [Heroku](https://www.heroku.com/)
- Click the 'New > Create new app' button
- Eventually choose your app name and your region, then click 'Create App'
- Go to the 'Settings' tab
- Click the 'Reveal Config Vars' button, and add vars `MAVEN_CUSTOM_OPTS`, `YACY_INIT_ADMIN_PASSWORD`, `YACY_PUBLIC_URL` with values filled as described in the previous paragraph
- Go to the 'Deploy' tab
- Choose 'GitHub' deployment method
- Enter your GitHub account information
- Select your YaCy repository clone and 'Connect'
- Choose either automatic or manual deploy, and click the deploy button
- If everything went fine, you can open YaCy search page with the 'View' button
## Technical details
### Heroku specific configuration files :
- [app.json](app.json) : used when deploying with the button
- [.env](.env) : set up environment variables for local heroku run
- [Procfile](Procfile) : contain main process description used to launch YaCy
### Custom maven options
With any of the deployment methods described, setting the option `-f libbuild/pom.xml -DskipTests=true` in the `MAVEN_CUSTOM_OPTS` environment variable is the minimum required for a successfull build and deploy. If not set, build will fail because missing dependent submodules from libbuild directory.
What'smore, the only way for other YaCy peers to reach a peer running on Heroku is to use the "dyno" public URL (in the form of your_app_name.herokuapp.com). This is why the configuration variable `YACY_PUBLIC_URL` as to be set, or else your YaCy peer will run in "junior" mode. This variable is used in the Procfile to customize the `staticIP` initial property in the [yacy.init](defaults/yacy.init) file at launch.
### HTTP local port
On heroku platform, you can not choose your application binding port. It is set by Heroku in the `PORT` environment variable, and at startup, the application has to bind to this port within 90 seconds. So YaCy has to bind to this port each time the "dyno" is started, and thus can not rely on its normal configuration file. This is done with the JVM system property `net.yacy.server.localPort` set in Procfile.
### Administrator password
If you wish to use YaCy administration features, you have to set an admin password, and authenticate with it. To generate the encoded password hash set in the config var `YACY_INIT_ADMIN_PASSWORD`, you can proceed as follow :
- either run a local YaCy peer, set your custom admin password in /ConfigAccounts_p.html, and retrieve it in DATA/SETTINGS/yacy.conf at key `adminAccountBase64MD5`
- OR run this command in YaCy source directory :
- `java -classpath target/classes:lib/* net.yacy.cora.order.Digest -strfhex "admin:The YaCy access is limited to administrators. If you don't know the password, you can change it using <yacy-home>/bin/passwd.sh <new-password>:[your_password]"`

@ -0,0 +1 @@
web: if [ "" != "$YACY_INIT_ADMIN_PASSWORD" ] ; then sed -i "/adminAccountBase64MD5=/c\adminAccountBase64MD5=$YACY_INIT_ADMIN_PASSWORD" defaults/yacy.init;fi && sed -i "/port = 8090/c\port = 80" defaults/yacy.init && if [ "" != "$YACY_PUBLIC_URL" ] ; then sed -i "/staticIP=/c\staticIP=$YACY_PUBLIC_URL" defaults/yacy.init;fi && java $JAVA_OPTS -Dnet.yacy.server.localPort="$PORT" -classpath target/classes:lib/* net.yacy.yacy

@ -1,4 +1,10 @@
== WHAT IS THIS? ==
# YaCy
[![Deploy to Docker Cloud](https://files.cloud.docker.com/images/deploy-to-dockercloud.svg)](https://cloud.docker.com/stack/deploy/?repo=https://github.com/yacy/yacy_search_server/tree/master/docker)
[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy)
## What is this?
YaCy is a search engine software. It takes a new approach to search
because it does not use a central server. Instead, its search results
come from a network of independent peers. In such a distributed network,
@ -34,24 +40,27 @@ set up their own node. More users are leading to higher index capacity
and better distributed indexing performance.
== LICENSE ==
## License
YaCy is published under the GPL v2
The source code is inside the release package (see /source and /htroot).
== WHERE IS THE DOCUMENTATION? ==
## Where is the documentation?
Documentation can be found at:
(Home Page) http://yacy.net/
(German Forum) http://forum.yacy.de/
(Wiki:de) http://www.yacy-websuche.de/wiki/index.php/De:Start
(Wiki:en) http://www.yacy-websearch.net/wiki/index.php/En:Start
(Tutorial Videos) http://yacy.net/en/Tutorials.html and http://yacy.net/de/Lehrfilme.html
- (Home Page) http://yacy.net/
- (German Forum) http://forum.yacy.de/
- (Wiki:de) http://www.yacy-websuche.de/wiki/index.php/De:Start
- (Wiki:en) http://www.yacy-websearch.net/wiki/index.php/En:Start
- (Tutorial Videos) http://yacy.net/en/Tutorials.html and http://yacy.net/de/Lehrfilme.html
Every of these locations has a (YaCy) search functionality which combines
all these locations into one search result.
== DEPENDENCIES? WHAT OTHER SOFTWARE DO I NEED? ==
## Dependencies? What other software do I need?
You need java 1.7 or later to run YaCy, nothing else.
Please download it from http://www.java.com
@ -62,7 +71,8 @@ NO OTHER SOFTWARE IS REQUIRED!
(you don't need apache, tomcat or mysql or whatever)
== HOW DO I START THIS SOFTWARE? ==
## How do I start this software?
Startup and Shutdown of YaCy:
- on GNU/Linux and OpenBSD:
@ -78,7 +88,8 @@ please use the Mac Application and start or stop it like any
other Mac Application (doubleclick to start)
== HOW DO I USE THIS SOFTWARE, WHERE IS THE ADMINISTRATION INTERFACE? ==
## How do I use this software, where is the administration interface?
YaCy is a build on a web server. After you started YaCy,
start your browser and open
@ -87,7 +98,8 @@ start your browser and open
There you can see your personal search and administration interface.
== WHAT IF I INSTALL YACY (HEADLESS) ON A SERVER ==
## What if I install YaCy (headless) on a server?
You can do that but YaCy authorizes users automatically if they
access the server from the localhost. After about 10 minutes a random
password is generated and then it is not possible to log in from
@ -99,22 +111,37 @@ http://<remote-server-address>:8090/ConfigAccounts_p.html
and set an administration account.
== CAN I RUN YACY IN A VIRTUAL MACHINE OR A CONTAINER ==
## Can I run YaCy in a virtual machine or a container?
YaCy runs fine in virtual machines managed by software such as VirtualBox or VMware.
Container technology may be more flexible and lightweight and also works fine with YaCy.
More details for YaCy with Docker [[docker/Readme.md|here]].
These technologies can either be deployed locally, on remote machines you own, or in the 'cloud'. Decide what fits the most your privacy requirements.
### Docker
Deploy easily YaCy on a Docker cloud provider of your choice (can be a machine you own) with the deploy button at the top of this page.
More details for YaCy with Docker in [docker/Readme.md](docker/Readme.md).
### Heroku
Deploy easily on [Heroku](https://www.heroku.com/) PaaS (Platform as a service) provider using the deploy button at the top.
More details for YaCy on Heroku in [Heroku.md](Heroku.md).
## Port 8090 is bad, people are not allowed to access that port
== PORT 8090 IS BAD, PEOPLE ARE NOT ALLOWED TO ACCESS THAT PORT ==
You can forward port 80 to 8090 with iptables:
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8090
On some operation systems, you must first enable access to the ports you are using like:
iptables -I INPUT -m tcp -p tcp --dport 8090 -j ACCEPT
== HOW CAN I SCALE THIS; HOW MUCH RAM IS NEEDED; DISK SPACE? ==
## How can I scale this; how much ram is needed; disk space?
YaCy can scale up to many millions of web pages in your own search index.
The default assignment of RAM is 600MB which is assigned to the java
process but not permanently used by it. The GC process will free the memory
@ -129,14 +156,15 @@ space i.e. setting the htcache space to a different size; to do that
open http://localhost:8090/ConfigHTCache_p.html and set a new size.
## Join the development!
== JOIN THE DEVELOPMENT! ==
YaCy was created with the help of many. About 30 programmers have helped,
a list of some of them can be seen here: http://yacy.net/en/Join.html
Please join us!
== HOW TO GET THE SOURCE CODE AND HOW TO COMPILE YACY YOURSELF? ==
## How to get the source code and how to compile YaCy yourself?
The source code is inside every YaCy release. You can also get YaCy
from https://github.com/yacy/yacy_search_server
Please clone our code and help with development!
@ -152,8 +180,8 @@ Compiling YaCy:
because the servlet pages are not compiled by the eclipse build process
after the dist prodecure, the release can be found in the RELEASE subdirectory
## Are there any APIs or how can I attach software at YaCy?
== ARE THERE ANY APIs OR HOW CAN I ATTACH SOFTWARE AT YACY? ==
There are many interfaces build-in in YaCy and they are all based on http/xml and
http/json. You can discover these interfaces if you notice the orange "API" icon in
the upper right of some web pages in the YaCy web interface. Just click on it and
@ -163,8 +191,8 @@ A different approach is the usage of the shell script provided in the /bin
subdirectory. The just call also the web interface pages. By cloning some of those
scripts you can create more shell api access methods yourself easily.
## Contact
== CONTACT ==
Our primary point of contact is the german forum at http://forum.yacy.net
There is also an english forum at http://www.yacy-forum.org
We encourage you to start a YaCy forum in your own language.

@ -0,0 +1,28 @@
{
"name": "YaCy",
"description": "Decentralized Web Search",
"keywords": [
"yacy",
"search",
"p2p",
"decentralized",
"java"
],
"website": "http://yacy.net",
"repository": "https://github.com/yacy/yacy_search_server",
"logo": "http://yacy.net/material/YaCyLogo2011.svg",
"env": {
"YACY_INIT_ADMIN_PASSWORD": {
"description": "Administrator initial password : please fill with a Base64 MD5 value",
"required": false
},
"YACY_PUBLIC_URL": {
"description": "External URL used by other peers to reach yours, required to run in 'Senior' mode. You chould set it with [your_app_name].herokuapp.com.",
"required": false
},
"MAVEN_CUSTOM_OPTS": {
"description": "Custom options required for maven build on Heroku.",
"value": "-f libbuild/pom.xml -DskipTests=true"
}
}
}

@ -1,7 +1,5 @@
# Yacy Docker image from latest sources
[![Deploy to Docker Cloud](https://files.cloud.docker.com/images/deploy-to-dockercloud.svg)](https://cloud.docker.com/stack/deploy/?repo=https://github.com/luccioman/yacy_search_server/tree/docker/docker)
## Supported tags and respective Dockerfiles
* latest (Dockerfile)
@ -111,7 +109,7 @@ OR
Create new container based on pulled image, using volume data from old container :
docker create --name [tmp-container_name] -p 8090:8090 --volumes-from=[container_name] luccioman/yacy:latest
docker create --name [tmp-container_name] -p 8090:8090 --volumes-from=[container_name] --log-opt max-size=100m --log-opt max-file=2 luccioman/yacy:latest
Stop old container :

@ -91,7 +91,7 @@
<dl>
<dt><label for="port">Peer Port: </label></dt>
<dd>
<input type="text" name="port" id="port" value="#[defaultPort]#" size="5" maxlength="5" />&nbsp;&nbsp;&nbsp;
<input type="text" name="port" id="port" value="#[defaultPort]#" size="5" maxlength="5" #(hasSystemDefinedPort)#::disabled="disabled"#(/hasSystemDefinedPort)#/> #(hasSystemDefinedPort)#::(Set by system property '#[systemProperty]#')#(/hasSystemDefinedPort)#&nbsp;&nbsp;&nbsp;
<input type="checkbox" name="withssl" id="withssl" #(withsslenabled)#::checked="checked"#(/withsslenabled)#>with SSL (https enabled#(withsslenabled)#:: on port <a href="Settings_p.html?page=ProxyAccess">#[sslport]#</a>#(/withsslenabled)#)
</dd>
#(upnp)#::<dt>

@ -133,7 +133,7 @@ public class ConfigBasic {
// check port and ssl connection
final boolean reconnect;
if (!(env.getConfigLong("port", port) == port) || env.getConfigBool("server.https", false) != ssl) {
if (!(env.getLocalPort() == port) || env.getConfigBool("server.https", false) != ssl) {
// validate port
final YaCyHttpServer theServerCore = env.getHttpServer();
env.setConfig("port", port);
@ -268,6 +268,8 @@ public class ConfigBasic {
// set default values
prop.putHTML("defaultName", sb.peers.mySeed().getName());
prop.put("hasSystemDefinedPort", env.getLocalPortSystemProperty() != null ? 1 : 0);
prop.put("hasSystemDefinedPort_systemProperty", serverSwitch.LOCAL_PORT_SYSTEM_PROPERTY);
prop.put("defaultPort", env.getLocalPort());
prop.put("withsslenabled", env.getConfigBool("server.https", false) ? 1 : 0);
lang = env.getConfig("locale.language", "default"); // re-assign lang, may have changed

@ -0,0 +1 @@
/target/

@ -329,6 +329,7 @@ However: if you fail to open a router port, you can nevertheless use YaCy with f
the only function that is missing is on the side of the other YaCy users because they cannot see your peer.==l'unique fonction manquante se situera du c&ocirc;t&eacute; des autres utilisateurs de YaCy car ils ne pourront pas voir votre noeud.
Your peer can be reached by other peers==Votre noeud est joignable par les autres noeuds du r&eacute;seau
Peer Port:==Port logiciel du noeud:
Set by system property==Configuré via la propriété système
with SSL==avec SSL
(https enabled#==(https activ&eacute;#
on port==sur le port

@ -1129,6 +1129,9 @@
<trans-unit id="9b86d79b" xml:space="preserve" approved="no" translate="yes">
<source>Peer Port:</source>
</trans-unit>
<trans-unit id="a24f437b" xml:space="preserve" approved="no" translate="yes">
<source>Set by system property</source>
</trans-unit>
<trans-unit id="c7d5ab92" xml:space="preserve" approved="no" translate="yes">
<source>with SSL</source>
</trans-unit>

@ -167,29 +167,6 @@
</configuration>
</plugin>
<!-- custom plugin to add YaCy release number from local Git clone
sets property <releaseNr>9nnn</releasNr> <DSTAMP>yyyyMMdd</DSTAMP>
! run sub project in libbuild to install the plugin ! -->
<plugin>
<groupId>net.yacy</groupId>
<artifactId>maven-plugin-gitrevisionnumber</artifactId>
<version>1.0</version>
<executions>
<execution>
<phase>initialize</phase>
<goals>
<goal>create</goal>
</goals>
<!-- optional parameter to set/change the propertyName
<configuration>
<branchPropertyName>branch</branchPropertyName>
<buildNumberPropertyName>releasNr</buildNumberPropertyName>
<commitDatePropertyName>DSTAMP</commitDatePropertyName>
</configuration>
-->
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>templating-maven-plugin</artifactId>
@ -241,6 +218,55 @@
</build>
<profiles>
<profile>
<!-- Active when .git directory is missing -->
<id>no-git</id>
<activation>
<file>
<missing>.git</missing>
</file>
</activation>
<properties>
<!-- fallback value consistent with ant build -->
<releaseNr>9000</releaseNr>
</properties>
</profile>
<profile>
<id>git-rev-number</id>
<!-- Active profile when .git folder exists -->
<activation>
<file>
<exists>.git</exists>
</file>
</activation>
<build>
<plugins>
<!-- custom plugin to add YaCy release number from local Git clone
sets property <releaseNr>9nnn</releasNr> <DSTAMP>yyyyMMdd</DSTAMP>
! run sub project in libbuild to install the plugin ! -->
<plugin>
<groupId>net.yacy</groupId>
<artifactId>maven-plugin-gitrevisionnumber</artifactId>
<version>1.0</version>
<executions>
<execution>
<phase>initialize</phase>
<goals>
<goal>create</goal>
</goals>
<!-- optional parameter to set/change the propertyName
<configuration>
<branchPropertyName>branch</branchPropertyName>
<buildNumberPropertyName>releasNr</buildNumberPropertyName>
<commitDatePropertyName>DSTAMP</commitDatePropertyName>
</configuration>
-->
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<!-- profile to create a release archive -->
<id>release-profile</id>

@ -58,6 +58,9 @@ import net.yacy.peers.Seed;
import net.yacy.search.SwitchboardConstants;
public class serverSwitch {
/** Key of system property defining locally open http port */
public static final String LOCAL_PORT_SYSTEM_PROPERTY = "net.yacy.server.localPort";
// configuration management
private final File configFile;
@ -217,18 +220,40 @@ public class serverSwitch {
return getConfigInt(key, dflt);
}
/**
* @return local http server port or null if system property is not defined
*/
public Integer getLocalPortSystemProperty() {
String systemDefinedPort = System.getProperty(LOCAL_PORT_SYSTEM_PROPERTY);
Integer localPort = null;
if(systemDefinedPort != null) {
try {
localPort = Integer.parseInt(systemDefinedPort);
} catch(NumberFormatException e) {
log.warn("System property " + LOCAL_PORT_SYSTEM_PROPERTY + " is not valid : it should be a integer.");
}
}
return localPort;
}
/**
* Wrapper for {@link #getConfigInt(String, int)} to have a more consistent
* API.
*
* Default value 8090 will be used if no value is found
*
* @return the local port of this system
* Default value 8090 will be used if no value is found in system properties and in configuration.
*
* @return the local http port of this system
* @see #getPublicPort(String, int)
*/
public int getLocalPort() {
/* A system property "net.yacy.server.localPort" may override configuration
* This is useful when running YaCy inside a container manager such as Heroku which decide which http port to use */
Integer localPort = getLocalPortSystemProperty();
if(localPort != null) {
return localPort;
}
return getConfigInt("port", 8090);
}

Loading…
Cancel
Save