This content originally appeared on DEV Community and was authored by Miku Chan
Overview of My Submission
Konohagakure Search is a Google Like Search Engine. It is built on the technology of django, dedicated mongo db server, python, scrapy, spacy, and nltk. Whenever a search query is given it first search the database if its present then it searches up the internet then saves the data and then it presents to you in a pretty way :)
Overview of the project
An efficient Search Engine with the following features:
It has distributed crawlers to crawl the private/air-gapped networks (data sources in these networks might include websites, files, databases) and works behind sections of networks secured by firewalls
It uses AI/ML/NLP/BDA for better search (queries and results) It abides by the secure coding practices (and SANS Top 25 web vulnerability mitigation techniques.)
It is a type of a search engine which takes keyword/expression as an input and crawls the web (internal network or internet) to get all the relevant information. The application dosen't have any vulnerabilities, it complies with OWASP Top 10 Outcome. This application scrape data, match it with the query and give out relevant/related information.
Note - Search as robust as possible (eg, it can correct misspelt query, suggest similar search terms, etc) be creative in your approach. Result obtained from search engine should displays the relevant matches as per search query/keyword along with the time taken by search engine to fetch that result.
Submission Category:
Choose Your Own Adventure
Link to Code
Dhruvacube
/
search-engine
Google Like Search Engine
Konohagakure Search
Overview of the project:
An efficient Search Engine with the following features
It has distributed crawlers to crawl the private/air-gapped networks (data sources in these networks might include websites, files, databases) and works behind sections of networks secured by firewalls
It uses AI/ML/NLP/BDA for better search (queries and results) It abides by the secure coding practices (and SANS Top 25 web vulnerability mitigation techniques.)
It is a type of a search engine which takes keyword/expression as an input and crawls the web (internal network or internet) to get all the relevant information. The application dosen't have any vulnerabilities, it complies with OWASP Top 10 Outcome. This application scrape data, match it with the query and give out relevant/related information.
Note - Search as robust as possible (eg, it can correct misspelt query, suggest similar search terms, etc) be creative in your approach. Result obtained from search engine shouldPackages that were required in making this project
- django
- dj-database-url
- celery
- django-redis
- django-htmlmin
- gunicorn
- redis
- hiredis
- djongo
- pymongo[srv]
- python-dotenv
- requests
- beautifulsoup4
- textblob
- spacy
- nltk
- spacy-alignments
- spacy-legacy
- spacy-loggers
- colorama
- transformers
- Scrapy
- cdx-toolkit
- uvicorn
- whitenoise
- colorlog
- uvloop
- spacy-transformers
- spacy-lookups-data
- django-celery-beat
- django-cors-headers
A YouTube Video Explaining all
This content originally appeared on DEV Community and was authored by Miku Chan
Miku Chan | Sciencx (2022-01-09T08:54:38+00:00) Konohagakure Search. Retrieved from https://www.scien.cx/2022/01/09/konohagakure-search/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.