Morioh
Login
Liya Gebre

Liya Gebre

3 years ago

A crawler based on Flask, Vue.js and BootstrapVue

This project is a website based on Flask and BootstrapVue that separates the front and back ends of the Hangzhou Electric Brush Questions ranking list.

add_notice

admin_index

admin_list

index

login

register

user_index

3.0 What have we added?

  • User login, modification, registration
  • User-defined webpage code
  • Administrator add and delete
  • Administrator administrator user and other administrators
  • One-click initialization and operation
  • Can run on Windows and Linux
  • Multi-process, high concurrency
  • Install directly using pip

2.0 original functions

  • Add user
  • Crawl user questions
  • Show announcement
  • Administrator login
  • Administrator manages users
  • Crawler status management

Environmental requirements

Operating system: either Windows or Linux. python3.x pip3 MySQL8.x

Quick start

Operating environment required for installation

## Install the required PIP library
pip3 install flask pymysql requests flask_cors
## Domestic users can use to add pip install time parameters, such as:
pip3 install -i https://mirrors.ustc.edu.cn/pypi/web/simple/ flask pymysql requests flask_cors
## Cloning project
git clone https://github.com/736248591/hdu_rank.git
## Into the project 
cd hdu_rank
 ## run Setup Assistant
python3 helper.py
## Start crawling process 
python3 hdu_crawl.py &

Note: Choose one of the following methods

1. Standalone operation

Note that Tornado cannot run in multiple processes under Windows.

## Note: Make sure to compile and install Python before installing a libffi-devel, or else be wrong here pip3 install. 
## Install the required PIP library
pip3 install tornado
## Domestic users can replace the above command with the following command
pip3 install -i https://mirrors.ustc.edu.cn/pypi/web/simple/ tornado
cd hdu_rank
 ## start Tornado 
python3 run_tornado.py

2. Run under NGINX

## Note: Make sure to compile and install Python before installing a libffi-devel, or else be wrong here pip3 install. 
## Install the required PIP library
pip3 install uWSGI
## Increase of user and group hdurank
/usr/sbin/groupadd hdurank
/usr/sbin/useradd -g hdurank hdurank
## Create a Domain Name
lnmp vhost add
## Follow the prompts to fill in your name and address local storage project. Note that the root directory of the website is filled in hdu_rank/static

vhost_add

## Edit configuration NGINX
vim /usr/local/nginx/conf/vhost/your domain name.conf
## Add the following at the same level of child location of the server. 
location ~ * /api/{
    include uwsgi_params ; 
    uwsgi_pass 127.0.0.1:5007 ;
}
## Restart nginx server
lnmp nginx reload
## Open a new screen, so after turning off the terminal program will not be closed
screen -R hdu_rank
## Start uwsgi server 
uwsgi --ini uwsgi.ini

3. mod_wsgi (Apache), other independent WSGI containers, FastCGI, CGI

These methods can be viewed in the official Chinese document of Flask , which will not be repeated here.

Advanced development

API interface

/api/get_rank Get ranking

Parameters: (None) Response data:

  {
      status: 操作状态 Boolean,
      notice: 公告 string,
      crawl_status: 爬虫状态 union("runnable","running","sleeping","stopped")
      users: 用户列表（当状态为True时，拥有这个字段）
      [
          {
              id: 用户ID unsigned int,
              uid 登录账号 string,
              class_name 班级 string,
              name 姓名 string,
              motto 格言 string,
              account 账号 string,
              solved_num 题数 int,
              status 状态 union("unchecked","fetching","active","disconnect")
          }
      ],
      user:{
          id 用户ID unsigned int,
          uid 登录账号 string,
          class 班级 string,
          name 姓名 string,
          motto 格言 string,
          account 账号 string,
          solved_num 题数 int,
          status 状态 union("unchecked","fetching","active"),
          html 自定义页面代码 string
      },
      admin:{
          id: 管理员ID int,
          uid: 管理员 string,
          is_super: 是否可以管理其他用户 bool,
      }
  }

/api/login User login

  {
      status: 操作状态 Boolean,
      mgs: 错误原因 (当状态为false时，拥有这个字段）string，
      // 当登录成功时候显示以下信息
      user:{
          id 用户ID unsigned int,
          uid 登录账号 string,
          class 班级 string,
          name 姓名 string,
          motto 格言 string,
          account 账号 string,
          solved_num 题数 int,
          status 状态 union("unchecked","fetching","active"),
          html 自定义页面代码 string
      },
      admin:{
          id: 管理员ID int,
          uid: 管理员 string,
          is_super: 是否可以管理其他用户 bool,
      }
  }

/api/put_user Add or modify user

Note: When modifying, you only need to submit the ID and modified fields. When adding users, no id is required. parameter:

  {
      status: 操作状态 Boolean,
      mgs: 错误原因 (当状态为false时，拥有这个字段）string
  }

/api/validate_user validation field

Description: Used to verify the fields when registering. parameter:

  {
      status: 操作状态 Boolean,
      mgs: 错误原因 (当状态为false时，拥有这个字段）string
  }

/api/logout can be used by logged out users and administrators.

Parameters: (None) Response data:

  {
      status: 操作状态 Boolean
  }

/api/remove_user delete user

Note: You must log in before you can use this interface. Users can only delete themselves, and only the administrator can delete any user. parameter:

Field nametype of dataDefaultsdescriptionidunsigned intUser IDResponse data:

  {
      status: 操作状态 Boolean,
      mgs: 错误原因 (当状态为false时，拥有这个字段）string
  }

/api/login_admin Administrator login

Note: If you do not fill in any information, the current login information will be returned. parameter:

  {
      status: 操作状态 Boolean,
      mgs: 错误原因 (当状态为false时，拥有这个字段）string,
      admin:{
          id: 管理员ID int,
          uid: 管理员 string,
          is_super: 是否可以管理其他用户 bool,
      }
  }

/api/list_admin Administrator list

Parameters: (None) Response data:

  {
      status: 操作状态 Boolean,
      mgs: 错误原因 (当状态为false时，拥有这个字段）string,
      admins:[
      	{
            id: 管理员ID int,
            uid: 管理员登录账号 int,
            is_super: 是否可以管理其他管理员 bool
        }
      ]
  }

/api/validate_admin Verify the administrator field

Description: Used to verify fields when adding an administrator. parameter:

  {
      status: 操作状态 Boolean,
      mgs: 错误原因 (当状态为false时，拥有这个字段）string
  }

/api/put_admin Add or modify administrator

Note: When modifying, you only need to submit the ID and modified fields. parameter:

  {
      status: 操作状态 Boolean,
      mgs: 错误原因 (当状态为false时，拥有这个字段）string,
  }

/api/remove_admin remove administrator

Note: You must log in before you can use this interface. Only super administrators can delete other administrators. parameter:

  {
      status: 操作状态 Boolean,
      mgs: 错误原因 (当状态为false时，拥有这个字段）string
  }

/api/crawl_start Start rolling

Note: Only the administrator can use this interface. Parameters: (None) Response data:

  {
      status: 操作状态 Boolean,
      mgs: 错误原因 (当状态为false时，拥有这个字段）string
  }

/api/crawl_stop Stop scrolling

Note: Only the administrator can use this interface. Parameters: (None) Response data:

  {
      status: 操作状态 Boolean,
      mgs: 错误原因 (当状态为false时，拥有这个字段）string
  }

/api/add_notice Add notification

Note: Only the administrator can use this interface. parameter:

  {
      status: 操作状态 Boolean,
      mgs: 错误原因 (当状态为false时，拥有这个字段）string
  }

Compile the client manually

Install Node.js and Yarn

cd hdu_rank
yarn global add @vue/cli
yarn install
yarn build

Update diary

September 10, 2020

Update dependencies

August 8, 2020

Update dependencies and recompile the client.

July 17, 2020

  • Update and recompile the client.

June 18, 2020 (3.0)

  • Significantly modify and refactor the front-end and back-end code.
  • Added user login, modification, and registration functions.
  • Added the function of user-defined webpage code.
  • Added administrator add and delete, administrator user and other administrator functions
  • One-click initialization and operation
  • Can run on Windows and Linux
  • Multi-process, high concurrency

March 17, 2020

  • Update the client dependency library and recompile the client.

December 27, 2019

  • Update the library.
  • Remove fibers dependency.

October 22, 2019

  • Increase the number of crawler retries to connect.
  • Update and recompile the client.

September 27, 2019

  • Modify the uwsgi server configuration.

  • Fixed some problems with crawler connection.

September 25, 2019

  • Removed the useless ribbon in the upper right corner, and added the github address to the footer to prevent the mobile page from becoming disgusting.
  • Modify the error of the page language tag.
  • Before the account legitimacy judgment, the judgment on the connection of Hangzhou Electric OJ was added.
  • Improve the display experience on the mobile terminal.

September 23, 2019

  • Modify the size of the name field in the Users table of the database from the length of 8 to 16.
  • Before adding an account, verify the account first.
  • Fixed the error that the diary cannot be written. The diary writing location is /tmp/uwsgi.pid
  • Update the dependency package and recompile the web client.

September 3, 2019

  • Modify the set remarks, and modify the main loop to cycle once every 30 minutes by default.

September 2, 2019

  • Update the dependency package and recompile the web client.
July 28, 2019
  • Modify the operation of refreshing the page without logging in when entering the administrator password and pressing Enter.
  • The announcement supports Markdown syntax highlighting.
  • All components have been updated.
  • A ribbon containing a link to the github project has been added to the upper right corner.
  • After the user is successfully added, the user information still exists in the form.
  • After the administrator logs in, the password is still stored in the form.
  • After reopening the webpage, the bulletin cannot be modified but can only be created.
  • Delete all console.log output in product mode.
May 31, 2019
  • Update all components to remove security risks.
  • The rich text editor has been removed.
  • Added the function of adding announcements.
  • Fix the problem of MySQL connection dropping when not in use for a long time.
May 31, 2019
  • Update AXIOS to 0.19.0 to fix its security vulnerabilities.
  • Incidentally, update other components.
May 26, 2019

  • Upgraded the version of the dependent package to solve the security risks.

  • Fixed the BUG of database file import.

  • Added the dependencies required by python3.7 in the documentation.

  • The administrator password was not created incorrectly.

  • Added website icon and title.

April 24, 2019
  • When the password does not exist, a new password will be automatically generated and saved in the admin.key file.
April 9, 2019

1.0 Initial release

Download Details:

Author: 736248591

Source Code: https://github.com/736248591/hdu_rank

#vuejs #vue #javascript

A crawler based on Flask, Vue.js and BootstrapVue
2.25 GEEK