Yacy Search Server
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
reger24 426c47013c Prepare to use Gradle dependencies while mimic current config 5 hours ago
.github fixed name 3 months ago
.gradle downgrading gradle to 6.9 3 months ago
.settings removed ymarks 4 months ago
LICENSES creating LICENSES directory 10 months ago
addon attempt to make a Mac Release using gradle 3 months ago
bin fixed apicall call method parameters 1 year ago
defaults lets SOLR merge bigger segments (up to 50GB) 3 months ago
dictionaries changed way to integrate dictionary files: 13 years ago
docker aded dockerfiles for 32 and 64 bit ARM/Raspberry Pi 1 year ago
examples move example code SearchClient out of yacycore package 6 years ago
gradle/wrapper downgrading gradle to 6.9 3 months ago
htroot Add Sorting functionality to Crawler Queue Table 1 week ago
kubernetes removed Debian and Red Hat build process 6 months ago
langdetect Use language-detection library for increased accuracy 7 years ago
lib updating slf4j 1.7.25 -> 1.7.32 1 month ago
libbuild Obsolete since replacement of Maven by Gradle build script 3 days ago
libt replace org.junit.Assert.assertThat by 1 year ago
locales removed ymarks 4 months ago
skins added stub of rc3assembly style 11 months ago
snap Added a stop command using the existing stop script to the snap package 3 years ago
source Prevent that YaCy always starts with a exception message on none Apple systems 8 hours ago
test fixed doku link 6 months ago
vocabularies moved yacy.logging to defaults according to request in 10 years ago
.checkstyle problems with code style 13 years ago
.classpath updating slf4j 1.7.25 -> 1.7.32 1 month ago
.dockerignore added a dockerignore file 1 year ago
.env Removed unnecessary variables for local heroku run 6 years ago
.gitignore add gradle to gitignore 6 months ago
.project removes some warnings: unused imports, params 1 year ago
.travis.yml removed Xms java memory startup parameter 6 months ago
AUTHORS added flori and me 14 years ago
CONTRIBUTING.md Add contributor guidelines; closes #214 3 years ago
COPYRIGHT updated copyright message; included LGPL for 'cora' and a warranty 9 years ago
Heroku.md Simplified Heroku variables configuration 6 years ago
NOTICE * updated jxpath to latest v1.3 13 years ago
Procfile Simplified Heroku variables configuration 6 years ago
README.md Update README.md 9 hours ago
app.json Updated YaCy home page embedded links from http to https scheme 4 years ago
build.gradle Prepare to use Gradle dependencies while mimic current config 5 hours ago
build.nsi upd windows installer script to Java 8 update 211 2 years ago
build.properties new version for solr 8 11 months ago
build.xml updating slf4j 1.7.25 -> 1.7.32 1 month ago
fixMacBuild.sh attempt to make a Mac Release using gradle 3 months ago
getWin32MaxHeap.bat update for memory observer algorithm 12 years ago
gpl.txt initial load with yacy 0.36 17 years ago
gradle.properties attempt to make a Mac Release using gradle 3 months ago
gradlew downgrading gradle to 6.9 3 months ago
gradlew.bat replaced maven with gradle 6 months ago
installYaCyWindowsService.bat removed Xms java memory startup parameter 6 months ago
killYACY.sh Fix for http://mantis.tokeek.de/view.php?id=432 8 years ago
lgpl21.txt migrated all my LGPL 3 -licensed files to the LGPL 2.1 because LGPL 3 is not compatible to the GPL 2 12 years ago
settings.gradle replaced maven with gradle 6 months ago
startYACY.bat removed Xms java memory startup parameter 6 months ago
startYACY.sh addressing better GC behavior after removing Xms with earlier heap increase strategy 6 months ago
startYACY_debug.bat removed Xms java memory startup parameter 6 months ago
stopYACY.bat upd classpath in batches (remove not necessary htroot) 6 years ago
stopYACY.sh fixed apicall call method parameters 1 year ago
uninstallYaCyWindowsService.bat added Windows Service installer 8 years ago
updateYACY.sh fixed apicall call method parameters 1 year ago
yacy-packages.readme bump to commons-codec-1.14, commons-compress-1.20, 2 years ago
yacy.yellow performance setting for remote indexing configuration and latest changes for 0.39 17 years ago

README.md

YaCy

Gitter Build Status

Deploy

What is this?

The YaCy search engine software provides results from a network of independent peers, instead of a central server. It is a distributed network where no single entity decides what to list or order it appears in.

User privacy is central to YaCy, and it runs on each user's computer, where search terms are hashed before they being sent to the network. Everyone can create their individual search indexes and rankings, and a truly customized search portal.

Each YaCy user is either part of a large search network (search indexes can be exchanged with other installation over a built-in peer-to-peer network protocol) or the user runs YaCy to produce a personal search portal that is either public or private.

YaCy search portals can also be placed in an intranet environment, making it a replacement for commercial enterprise search solutions. A network scanner makes it easy to discover all available HTTP, FTP and SMB servers.

To create a web index, YaCy has a web crawler for everybody, free of censorship and central data retention:

  • Search the web (automatically using all other YaCy peers)
  • Co-operative crawling; support for other crawlers
  • Intranet indexing and search
  • Set up your own search portal
  • All users have equal rights
  • Comprehensive concept to anonymise the users' index

To be able to perform a search using the YaCy network, every user has to set up their own node. More users means higher index capacity and better distributed indexing performance.

License

This project is available as open source under the terms of the GPL 2.0 Or later. However, some elements are being licensed under GNU Lesser General Public License. For accurate information, please check individual files. As well as for accurate information regarding copyrights. The (GPLv2+) source code used to build YaCy is distributed with the package (in /source and /htroot).

Where is the documentation?

All these have (YaCy) search functionality combining all these locations into one search result.

Dependencies? What other software do I need?

You need Java 1.8 or later to run YaCy. (No Apache, Tomcat or MySQL or anything else)

YaCy also runs on IcedTea 3. See https://icedtea.classpath.org

Start and stop it

Startup and shutdown:

  • GNU/Linux and OpenBSD:

    • Start by running ./startYACY.sh
    • Stop by running ./stopYACY.sh
  • Windows:

    • Start by double-clicking startYACY.bat
    • Stop by double-clicking stopYACY.bat
  • macOS: Please use the Mac app and start or stop it like any other program (double-click to start)

The administration interface

A web server us brought up after starting YaCy. Open this URL in your web-browser:

http://localhost:8090

This presents you with the personal search and administration interface.

(Headless) YaCy server installation

YaCy will authorize users automatically if they access the server from its localhost. After about 10 minutes a random password is generated and then it is no longer possible to log in from a remote location. If you install YaCy on a server that is not your workstation you must set an admin account immediately after the first start-up. Open:

http://<remote-server-address>:8090/ConfigAccounts_p.html

and set an admin account.

YaCy in a virtual machine or a container

Use virtualization software like VirtualBox or VMware.

The following container technologies can deployed locally, on remote machines you own, or in the 'cloud' using a provider by clicking "Deploy" at the top of the page:

Docker

More details in the docker/Readme.md.

Heroku

PaaS (Platform as a service) More details in Heroku.md.

Port 8090 is bad, people are not allowed to access that port

You can forward port 80 to 8090 with iptables:

iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8090

On some operation systems, access to the ports you are using must be granted first:

iptables -I INPUT -m tcp -p tcp --dport 8090 -j ACCEPT

Scaling, RAM and disk space

You can have many millions web pages in your own search index. By default 600MB RAM is available to the Java process. The GC process will free the memory once in a while. If you have less than 100000 pages you could try 200MB till you hit 1 million. Here you can adjust it. Several million web pages may use several GB of disk space, but you can adjust it Here to fit your needs.

Help develop YaCy

Join the large number of contributors that make YaCy what it is; community software.

To start developing YaCy in Eclipse:

  • Clone https://github.com/yacy/yacy_search_server.git
  • File → Import as Git → Projects from Git → Existing local repository
  • → Add → Your Git clone of yacy_search_server
  • "Import existing Eclipse projects" → Finish
  • Run → External Tools → External Tools Configuration → Double-click "Ant Build"
  • → Name: "YaCy Build" → Buildfile: Browse Workspace → build.xml → Run
  • In the Package Explorer, right-click on YaCy → Run as → Java Application -> Select "yacy - net.yacy" -> OK

To join our development community, got to https://searchlab.eu

Send pull requests to https://github.com/yacy/yacy_search_server

Compile from source

The source code is bundled with every YaCy release. You can also get YaCy from https://github.com/yacy/yacy_search_server by cloning the repository.

git clone https://github.com/yacy/yacy_search_server

Compiling YaCy:

  • You need Java 1.8 or later and Apache Ant.
  • Compile: ant clean all - then you can ./startYACY.sh or ./startYACY.bat.
  • Create a release tarball: ant dist.
  • Create a macOS release: ant distMacApp (only works on macOS).
  • Create a Debian release: ant deb.
  • Work with Eclipse: Within Eclipse you also need to start the ant build process. because the servlet pages are not compiled by the Eclipse build process. after the dist procedure, the release can be found in the RELEASE subdirectory.

APIs and attaching software

YaCy has many built-in interfaces, and they are all based on HTTP/XML and HTTP/JSON. You can discover these interfaces if you notice the orange "API" icon in the upper right corner of some web pages in the YaCy web interface. Click it and you will see the XML/JSON version of the respective webpage. You can also use the shell script provided in the /bin subdirectory. The shell scripts also calls the YaCy web interface. By cloning some of those scripts you can easily create more shell API access methods.

Contact

Visit the international YaCy forum where you can start a discussion there in your own language.

Questions and requests for paid customization and integration into enterprise solutions. can be sent to the maintainer, Michael Christen per e-mail (at mc@yacy.net) with a meaningful subject including the word 'YaCy' to prevent it getting stuck in the spam filter.

  • Michael Peter Christen