/
gtr
/
cmc-py
Обзор
Документация
Войти
/
gtr
/
cmc-py
Код
Запросы
0
Задачи
Вики
Пакеты
0
Релизы
0
Аналитика
Безопасность
main
docs/modules/base.html
310 строк
18 KB
Devansh3712
Create `cryptocurrency.price_prediction` module
23 фев 2022, 09:53
23 фев 2022, 09:53
75b8cc8
Код
Авторство
О чём код?
<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1" /> <meta name="generator" content="pdoc 0.10.0" /> <title>cmc.modules.base API documentation</title> <meta name="description" content="Module for storing settings for Selenium and requests used by other cmc-py modules …" /> <link rel="preload stylesheet" as="style" href="https://cdnjs.cloudflare.com/ajax/libs/10up-sanitize.css/11.0.1/sanitize.min.css" integrity="sha256-PK9q560IAAa6WVRRh76LtCaI8pjTJ2z11v0miyNNjrs=" crossorigin> <link rel="preload stylesheet" as="style" href="https://cdnjs.cloudflare.com/ajax/libs/10up-sanitize.css/11.0.1/typography.min.css" integrity="sha256-7l/o7C8jubJiy74VsKTidCy1yBkRtiUGbVkYBylBqUg=" crossorigin> <link rel="stylesheet preload" as="style" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.1.1/styles/github.min.css" crossorigin> <style>:root{--highlight-color:#fe9}.flex{display:flex !important}body{line-height:1.5em}#content{padding:20px}#sidebar{padding:30px;overflow:hidden}#sidebar > *:last-child{margin-bottom:2cm}.http-server-breadcrumbs{font-size:130%;margin:0 0 15px 0}#footer{font-size:.75em;padding:5px 30px;border-top:1px solid #ddd;text-align:right}#footer p{margin:0 0 0 1em;display:inline-block}#footer p:last-child{margin-right:30px}h1,h2,h3,h4,h5{font-weight:300}h1{font-size:2.5em;line-height:1.1em}h2{font-size:1.75em;margin:1em 0 .50em 0}h3{font-size:1.4em;margin:25px 0 10px 0}h4{margin:0;font-size:105%}h1:target,h2:target,h3:target,h4:target,h5:target,h6:target{background:var(--highlight-color);padding:.2em 0}a{color:#058;text-decoration:none;transition:color .3s ease-in-out}a:hover{color:#e82}.title code{font-weight:bold}h2[id^="header-"]{margin-top:2em}.ident{color:#900}pre code{background:#f8f8f8;font-size:.8em;line-height:1.4em}code{background:#f2f2f1;padding:1px 4px;overflow-wrap:break-word}h1 code{background:transparent}pre{background:#f8f8f8;border:0;border-top:1px solid #ccc;border-bottom:1px solid #ccc;margin:1em 0;padding:1ex}#http-server-module-list{display:flex;flex-flow:column}#http-server-module-list div{display:flex}#http-server-module-list dt{min-width:10%}#http-server-module-list p{margin-top:0}.toc ul,#index{list-style-type:none;margin:0;padding:0}#index code{background:transparent}#index h3{border-bottom:1px solid #ddd}#index ul{padding:0}#index h4{margin-top:.6em;font-weight:bold}@media (min-width:200ex){#index .two-column{column-count:2}}@media (min-width:300ex){#index .two-column{column-count:3}}dl{margin-bottom:2em}dl dl:last-child{margin-bottom:4em}dd{margin:0 0 1em 3em}#header-classes + dl > dd{margin-bottom:3em}dd dd{margin-left:2em}dd p{margin:10px 0}.name{background:#eee;font-weight:bold;font-size:.85em;padding:5px 10px;display:inline-block;min-width:40%}.name:hover{background:#e0e0e0}dt:target .name{background:var(--highlight-color)}.name > span:first-child{white-space:nowrap}.name.class > span:nth-child(2){margin-left:.4em}.inherited{color:#999;border-left:5px solid #eee;padding-left:1em}.inheritance em{font-style:normal;font-weight:bold}.desc h2{font-weight:400;font-size:1.25em}.desc h3{font-size:1em}.desc dt code{background:inherit}.source summary,.git-link-div{color:#666;text-align:right;font-weight:400;font-size:.8em;text-transform:uppercase}.source summary > *{white-space:nowrap;cursor:pointer}.git-link{color:inherit;margin-left:1em}.source pre{max-height:500px;overflow:auto;margin:0}.source pre code{font-size:12px;overflow:visible}.hlist{list-style:none}.hlist li{display:inline}.hlist li:after{content:',\2002'}.hlist li:last-child:after{content:none}.hlist .hlist{display:inline;padding-left:1em}img{max-width:100%}td{padding:0 .5em}.admonition{padding:.1em .5em;margin-bottom:1em}.admonition-title{font-weight:bold}.admonition.note,.admonition.info,.admonition.important{background:#aef}.admonition.todo,.admonition.versionadded,.admonition.tip,.admonition.hint{background:#dfd}.admonition.warning,.admonition.versionchanged,.admonition.deprecated{background:#fd4}.admonition.error,.admonition.danger,.admonition.caution{background:lightpink}</style> <style media="screen and (min-width: 700px)">@media screen and (min-width:700px){#sidebar{width:30%;height:100vh;overflow:auto;position:sticky;top:0}#content{width:70%;max-width:100ch;padding:3em 4em;border-left:1px solid #ddd}pre code{font-size:1em}.item .name{font-size:1em}main{display:flex;flex-direction:row-reverse;justify-content:flex-end}.toc ul ul,#index ul{padding-left:1.5em}.toc > ul > li{margin-top:.5em}}</style> <style media="print">@media print{#sidebar h1{page-break-before:always}.source{display:none}}@media print{*{background:transparent !important;color:#000 !important;box-shadow:none !important;text-shadow:none !important}a[href]:after{content:" (" attr(href) ")";font-size:90%}a[href][title]:after{content:none}abbr[title]:after{content:" (" attr(title) ")"}.ir a:after,a[href^="javascript:"]:after,a[href^="#"]:after{content:""}pre,blockquote{border:1px solid #999;page-break-inside:avoid}thead{display:table-header-group}tr,img{page-break-inside:avoid}img{max-width:100% !important}@page{margin:0.5cm}p,h2,h3{orphans:3;widows:3}h1,h2,h3,h4,h5,h6{page-break-after:avoid}}</style> <script defer src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.1.1/highlight.min.js" integrity="sha256-Uv3H6lx7dJmRfRvH8TH6kJD1TSK1aFcwgx+mdg3epi8=" crossorigin></script> <script>window.addEventListener('DOMContentLoaded', () => hljs.initHighlighting())</script> </head> <body> <main> <article id="content"> <header> <h1 class="title">Module <code>cmc.modules.base</code></h1> </header> <section id="section-intro"> <p>Module for storing settings for Selenium and requests used by other cmc-py modules.</p> <p>A random User-Agent and a proxy is used for requests session and Selenium driver in order to circumvent an IP ban. Data is scraped through Selenium (to load JavaScript components) and BeautifulSoup (to parse website data).</p> <details class="source"> <summary> <span>Expand source code</span> </summary> <pre><code class="python">#!/usr/bin/env python """Module for storing settings for Selenium and requests used by other cmc-py modules. A random User-Agent and a proxy is used for requests session and Selenium driver in order to circumvent an IP ban. Data is scraped through Selenium (to load JavaScript components) and BeautifulSoup (to parse website data). """ import os import random import re import time from typing import Dict, Optional import requests from requests.structures import CaseInsensitiveDict from selenium import webdriver from selenium.webdriver.chrome.service import Service from selenium.webdriver.common.proxy import Proxy, ProxyType from webdriver_manager.chrome import ChromeDriverManager from cmc.resources.user_agents import user_agents from cmc.utils.exceptions import ProxyTimeOut, InvalidProxy class CMCBaseClass: """Class for basic Selenium and requests settings for cmc-py modules. Sets up a random User-Agent and a random proxy each time the class is called. """ def __init__(self, proxy: Optional[str]): """ Args: proxy (Optional[str]): Proxy to be used for Selenium and requests Session. """ self.current_dir = os.path.dirname(os.path.realpath(__file__)) self.parent_dir = os.path.dirname(self.current_dir) self.cmc_url = "https://coinmarketcap.com" self.__proxy_url_1 = "https://public.freeproxyapi.com/api/Proxy/ProxyByType/0/4" self.__proxy_url_2 = "http://pubproxy.com/api/proxy?https=true" self.headers = CaseInsensitiveDict({"User-Agent": self.__get_random_user_agent}) self.session = requests.Session() self.session.headers = self.headers self.proxy: str = self.__get_proxy if proxy is None else proxy self.__check_proxy self.session.proxies = {"https": self.proxy} self.selenium_proxy = Proxy() self.selenium_proxy.proxy_type = ProxyType.MANUAL self.selenium_proxy.http_proxy = ( self.selenium_proxy.socks_proxy ) = self.selenium_proxy.ssl_proxy = self.proxy self.driver_options = webdriver.ChromeOptions() self.driver_options.Proxy = self.selenium_proxy self.driver_options.add_argument("headless") self.driver_options.add_argument("--no-sandbox") self.driver_options.add_argument("--log-level=3") self.driver_options.add_argument("ignore-certificate-errors") self.driver_options.add_experimental_option( "excludeSwitches", ["enable-logging"] ) self.service = Service(ChromeDriverManager(log_level=0).install()) @property def __get_proxy(self) -> str: """Fetch a random HTTPS proxy for using with Selenium. Raises: ProxyTimeOut: Raised when a proxy cannot be fetched from the API. Returns: str: Fetched proxy from the API. """ try: result = self.session.get(self.__proxy_url_1).json() proxy: str = result["host"] + ":" + str(result["port"]) time.sleep(1.5) return proxy except: try: result = self.session.get(self.__proxy_url_2).json() # type: ignore proxy: str = result["data"][0]["ipPort"] # type: ignore time.sleep(1.5) return proxy except: raise ProxyTimeOut @property def __get_random_user_agent(self) -> str: """Fetch a random User-Agent for using with requests Session. Returns: str: User-Agent for requests Session header. """ result: str = random.choice(user_agents) return result @property def __check_proxy(self) -> None: """Check whether the proxy (IP:Port) is valid or not. Raises: InvalidProxy: Raised if the proxy is not valid. """ regex = re.compile( r"^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]):[0-9]+$", re.IGNORECASE, ) if not regex.search(self.proxy): raise InvalidProxy(self.proxy) return</code></pre> </details> </section> <section> </section> <section> </section> <section> </section> <section> <h2 class="section-title" id="header-classes">Classes</h2> <dl> <dt id="cmc.modules.base.CMCBaseClass"><code class="flex name class"> <span>class <span class="ident">CMCBaseClass</span></span> <span>(</span><span>proxy: Optional[str])</span> </code></dt> <dd> <div class="desc"><p>Class for basic Selenium and requests settings for cmc-py modules. Sets up a random User-Agent and a random proxy each time the class is called.</p> <h2 id="args">Args</h2> <dl> <dt><strong><code>proxy</code></strong> : <code>Optional[str]</code></dt> <dd>Proxy to be used for Selenium and requests Session.</dd> </dl></div> <details class="source"> <summary> <span>Expand source code</span> </summary> <pre><code class="python">class CMCBaseClass: """Class for basic Selenium and requests settings for cmc-py modules. Sets up a random User-Agent and a random proxy each time the class is called. """ def __init__(self, proxy: Optional[str]): """ Args: proxy (Optional[str]): Proxy to be used for Selenium and requests Session. """ self.current_dir = os.path.dirname(os.path.realpath(__file__)) self.parent_dir = os.path.dirname(self.current_dir) self.cmc_url = "https://coinmarketcap.com" self.__proxy_url_1 = "https://public.freeproxyapi.com/api/Proxy/ProxyByType/0/4" self.__proxy_url_2 = "http://pubproxy.com/api/proxy?https=true" self.headers = CaseInsensitiveDict({"User-Agent": self.__get_random_user_agent}) self.session = requests.Session() self.session.headers = self.headers self.proxy: str = self.__get_proxy if proxy is None else proxy self.__check_proxy self.session.proxies = {"https": self.proxy} self.selenium_proxy = Proxy() self.selenium_proxy.proxy_type = ProxyType.MANUAL self.selenium_proxy.http_proxy = ( self.selenium_proxy.socks_proxy ) = self.selenium_proxy.ssl_proxy = self.proxy self.driver_options = webdriver.ChromeOptions() self.driver_options.Proxy = self.selenium_proxy self.driver_options.add_argument("headless") self.driver_options.add_argument("--no-sandbox") self.driver_options.add_argument("--log-level=3") self.driver_options.add_argument("ignore-certificate-errors") self.driver_options.add_experimental_option( "excludeSwitches", ["enable-logging"] ) self.service = Service(ChromeDriverManager(log_level=0).install()) @property def __get_proxy(self) -> str: """Fetch a random HTTPS proxy for using with Selenium. Raises: ProxyTimeOut: Raised when a proxy cannot be fetched from the API. Returns: str: Fetched proxy from the API. """ try: result = self.session.get(self.__proxy_url_1).json() proxy: str = result["host"] + ":" + str(result["port"]) time.sleep(1.5) return proxy except: try: result = self.session.get(self.__proxy_url_2).json() # type: ignore proxy: str = result["data"][0]["ipPort"] # type: ignore time.sleep(1.5) return proxy except: raise ProxyTimeOut @property def __get_random_user_agent(self) -> str: """Fetch a random User-Agent for using with requests Session. Returns: str: User-Agent for requests Session header. """ result: str = random.choice(user_agents) return result @property def __check_proxy(self) -> None: """Check whether the proxy (IP:Port) is valid or not. Raises: InvalidProxy: Raised if the proxy is not valid. """ regex = re.compile( r"^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]):[0-9]+$", re.IGNORECASE, ) if not regex.search(self.proxy): raise InvalidProxy(self.proxy) return</code></pre> </details> <h3>Subclasses</h3> <ul class="hlist"> <li><a title="cmc.modules.cryptocurrency.currency.CryptoCurrency" href="cryptocurrency/currency.html#cmc.modules.cryptocurrency.currency.CryptoCurrency">CryptoCurrency</a></li> <li><a title="cmc.modules.cryptocurrency.gainers.TopGainers" href="cryptocurrency/gainers.html#cmc.modules.cryptocurrency.gainers.TopGainers">TopGainers</a></li> <li><a title="cmc.modules.cryptocurrency.losers.TopLosers" href="cryptocurrency/losers.html#cmc.modules.cryptocurrency.losers.TopLosers">TopLosers</a></li> <li><a title="cmc.modules.cryptocurrency.most_visited.MostVisited" href="cryptocurrency/most_visited.html#cmc.modules.cryptocurrency.most_visited.MostVisited">MostVisited</a></li> <li><a title="cmc.modules.cryptocurrency.price_prediction.PricePrediction" href="cryptocurrency/price_prediction.html#cmc.modules.cryptocurrency.price_prediction.PricePrediction">PricePrediction</a></li> <li><a title="cmc.modules.cryptocurrency.ranking.Ranking" href="cryptocurrency/ranking.html#cmc.modules.cryptocurrency.ranking.Ranking">Ranking</a></li> <li><a title="cmc.modules.cryptocurrency.recently_added.RecentlyAdded" href="cryptocurrency/recently_added.html#cmc.modules.cryptocurrency.recently_added.RecentlyAdded">RecentlyAdded</a></li> <li><a title="cmc.modules.cryptocurrency.trending.Trending" href="cryptocurrency/trending.html#cmc.modules.cryptocurrency.trending.Trending">Trending</a></li> <li><a title="cmc.modules.exchange.derivatives.Derivatives" href="exchange/derivatives.html#cmc.modules.exchange.derivatives.Derivatives">Derivatives</a></li> <li><a title="cmc.modules.exchange.dex.Dex" href="exchange/dex.html#cmc.modules.exchange.dex.Dex">Dex</a></li> <li><a title="cmc.modules.exchange.exchange.Exchange" href="exchange/exchange.html#cmc.modules.exchange.exchange.Exchange">Exchange</a></li> <li><a title="cmc.modules.exchange.lending.Lending" href="exchange/lending.html#cmc.modules.exchange.lending.Lending">Lending</a></li> <li><a title="cmc.modules.exchange.spot.Spot" href="exchange/spot.html#cmc.modules.exchange.spot.Spot">Spot</a></li> <li><a title="cmc.modules.nft.collection.NFTRanking" href="nft/collection.html#cmc.modules.nft.collection.NFTRanking">NFTRanking</a></li> <li><a title="cmc.modules.nft.upcoming.UpcomingSale" href="nft/upcoming.html#cmc.modules.nft.upcoming.UpcomingSale">UpcomingSale</a></li> </ul> </dd> </dl> </section> </article> <nav id="sidebar"> <h1>Index</h1> <div class="toc"> <ul></ul> </div> <ul id="index"> <li><h3>Super-module</h3> <ul> <li><code><a title="cmc.modules" href="index.html">cmc.modules</a></code></li> </ul> </li> <li><h3><a href="#header-classes">Classes</a></h3> <ul> <li> <h4><code><a title="cmc.modules.base.CMCBaseClass" href="#cmc.modules.base.CMCBaseClass">CMCBaseClass</a></code></h4> </li> </ul> </li> </ul> </nav> </main> <footer id="footer"> <p>Generated by <a href="https://pdoc3.github.io/pdoc" title="pdoc: Python API documentation generator"><cite>pdoc</cite> 0.10.0</a>.</p> </footer> </body> </html>