Introduce to my Taiwan COVID-19 news website and crawler

Instructions

In Taiwan, it’s COVID-19 outbreak on May 13 and there’re about 100 local cases every day. And it’s zero local case since October.

At that moment, I’m really concerned about daily local cases so I develop this project and publis…


This content originally appeared on DEV Community and was authored by peter279k

Instructions

In Taiwan, it's COVID-19 outbreak on May 13 and there're about 100 local cases every day. And it's zero local case since October.

At that moment, I'm really concerned about daily local cases so I develop this project and publish it to the GitHub. I use the cron job feature to fetch the latest COVID-19 news and publish my new web page during GitHub action running.

My Workflow

My completed workflow files are available here and it includes following YAML files:

  • fetcher.yml, this YAML file is for fetching latest COVID-19 news and store them to be JSON file. This web crawler is written in Python. And It will be run every hour.
  • web_build.yml, this YAML file is for publishing new website and loading new JSON file. This website is written JavaScript with Vue.js framework. And it will be run every seventy minutes.

Submission Category:

This category is for DIY-Deployments and I hope developers love this and can be referenced :-).

YAML files and link to repository

Here are repository preview and completed codes about above two YAML files:


  • fetcher.yml
name: Update CSV Dataset Records from Taiwan CDC website

on:
  schedule:
    - cron: '0 * * * *'
  workflow_dispatch:

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Get working copy
        uses: actions/checkout@master
        with:
          fetch-depth: 1
      - name: Set up Python 3.8
        uses: actions/setup-python@v2
        with:
          python-version: '3.8'
      - name: Install Requirements
        run: pip install -r requirements.txt
      - name: Update COVD-19 Labs
        run: python Fetcher/LabFetcher.py
      - name: Update MOHW RSS News
        run: python Fetcher/MohwRssFetcher.py
      - name: Update COVID-19 global cases and deaths
        run: python Fetcher/GlobalCasesFetcher.py
      - name: Update COVID-19 TW datasets
        run: python Fetcher/DailyCovid19TW.py
      - name: Copy JSON files to assets folder
        run: cp datasets/*.json web/src/assets/
      - name: Commit and push if it changed
        run: |
          git config user.name "peter279k"
          git config user.email "peter279k@gmail.com"
          git add -A
          timestamp=$(date -u)
          git commit -m "Last Commit: ${timestamp}(TW)" || exit 0
          git push origin master
  • web_build.yml
name: 'Web Build and Deploy'

on:
  schedule:
    - cron: '10 * * * *'
  workflow_dispatch:
  push:
    branches:
      - main

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎️
        uses: actions/checkout@v2.3.1
        with:
          persist-credentials: false

      - name: Install and Build 🔧
        working-directory: web
        run: |
          cp .env.example .env
          export VUE_APP_MAPKEY=${{ secrets.VUE_APP_MAPKEY }}
          sed -i -e "s/MAP_KEY/$VUE_APP_MAPKEY/g" ./.env
          npm install
          npm run build
          cd dist
          cp index.html 404.html

      - name: Deploy 🚀
        uses: JamesIves/github-pages-deploy-action@3.6.2
        with:
          GIT_CONFIG_NAME: "peter279k"
          GIT_CONFIG_EMAIL: "peter279k@gmail.com"
          SINGLE_COMMIT: true
          GITHUB_TOKEN: ${{ secrets.TOKEN }}
          BRANCH: gh-pages
          FOLDER: web//dist

Additional Resources / Info

Here are some useful references about GitHub action workflows:


This content originally appeared on DEV Community and was authored by peter279k


Print Share Comment Cite Upload Translate Updates
APA

peter279k | Sciencx (2021-12-07T15:27:28+00:00) Introduce to my Taiwan COVID-19 news website and crawler. Retrieved from https://www.scien.cx/2021/12/07/introduce-to-my-taiwan-covid-19-news-website-and-crawler/

MLA
" » Introduce to my Taiwan COVID-19 news website and crawler." peter279k | Sciencx - Tuesday December 7, 2021, https://www.scien.cx/2021/12/07/introduce-to-my-taiwan-covid-19-news-website-and-crawler/
HARVARD
peter279k | Sciencx Tuesday December 7, 2021 » Introduce to my Taiwan COVID-19 news website and crawler., viewed ,<https://www.scien.cx/2021/12/07/introduce-to-my-taiwan-covid-19-news-website-and-crawler/>
VANCOUVER
peter279k | Sciencx - » Introduce to my Taiwan COVID-19 news website and crawler. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2021/12/07/introduce-to-my-taiwan-covid-19-news-website-and-crawler/
CHICAGO
" » Introduce to my Taiwan COVID-19 news website and crawler." peter279k | Sciencx - Accessed . https://www.scien.cx/2021/12/07/introduce-to-my-taiwan-covid-19-news-website-and-crawler/
IEEE
" » Introduce to my Taiwan COVID-19 news website and crawler." peter279k | Sciencx [Online]. Available: https://www.scien.cx/2021/12/07/introduce-to-my-taiwan-covid-19-news-website-and-crawler/. [Accessed: ]
rf:citation
» Introduce to my Taiwan COVID-19 news website and crawler | peter279k | Sciencx | https://www.scien.cx/2021/12/07/introduce-to-my-taiwan-covid-19-news-website-and-crawler/ |

Please log in to upload a file.




There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.