Oliver xu's Blog
0%

Paperspace白嫖指南

发表于 更新于 分类于
本文字数： 8.6k 阅读时长 ≈ 8 分钟

由于Colab的使用体验不是很好，所以，在查资料的过程中发现还有一个平台paperspace.com，于是尝试用了一个月，发现还不错，分享一下使用体验和写的一些自动化小脚本。

选择Paperspace的原因

优点：

  1. 身边没电脑（疫情被封着，电脑在实验室呢，出问题远程连不上了）
  2. 相比于Colab，该平台用的是nvidia-docker相关技术，持续化存储数据，不需要像Colab那样每次挂载Google driver，而且磁盘速度很快。
  3. 性能还凑合，8G显存的GPU，功率120W，凑合用吧。
  4. 能够使用自定义的docker（很赞）

缺点：

  1. 尝试过搭建代理，把SSH通过cloudflare tunnel转发出来，被封号。

性能配置：

使用方式和Google Colab基本类似，都是免费提供GPU平台，免费的配置是：

  • CPU: 8 cores
  • GPU: Quadro M4000
  • RAM: 32GB

性能还是可以的，就这个配置，肯定比很多实验室的普通办公电脑强多了，虽然比不上我的工作电脑，但是不要钱，不要电费，没有噪声污染，还是挺爽的

Docker实例

https://github.com/olixu/paper_code_env/blob/main/Dockerfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
# 基于最新的debian版本构建
FROM continuumio/anaconda3

# # 更新系统，包括python3
RUN apt -y update && \
    apt -y upgrade && \
    apt install -y python3-pip && pip3 install --upgrade pip && \
    apt install -y wget git htop vim zsh texlive texlive-latex-extra texlive-fonts-recommended dvipng cm-super && \
    wget https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh -O - | sh && \
#     chsh -s /bin/zsh root && \
    rm -rf /var/lib/apt/lists/*

# # #  安装jupyter lab等python的包
# RUN conda install -y pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch && \
# #     conda install -y jupyterlab && \
#     pip3 install --no-cache-dir cvxpy cvxpylayers matplotlib pandas jupyterlab-language-pack-zh-CN && \
#     conda clean -y -a     
   
# # 安装jupyter lab extensions
# RUN conda install -y pytorch torchvision torchaudio cudatoolkit=11.2 -c pytorch && \
RUN conda install -y pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.2 -c pytorch -c conda-forge && \
    conda install -y -c conda-forge nodejs jupyterlab-lsp python-lsp-server ipympl jupyterlab-drawio && \
    conda config --add channels https://conda.anaconda.org/gurobi && \
    conda install -y gurobi && \
    pip3 install --no-cache-dir jupyterlab-topbar jupyterlab-system-monitor lckr-jupyterlab-variableinspector  && \
    pip3 install --no-cache-dir SciencePlots tensorboard tensorwaves[jax] symengine casadi stable-baselines3 gym cvxpy cvxpylayers matplotlib pandas jupyterlab-language-pack-zh-CN && \
    pip3 install --no-cache-dir attrs==21.4.0 && \
    pip3 install --no-cache-dir --upgrade jax[cuda] jaxlib -f https://storage.googleapis.com/jax-releases/jax_releases.html && \
    conda clean -y -a 

# 使用8888端口访问
EXPOSE 8888


# The main purpose of a CMD is to provide defaults for an executing container
# This CMD opens the jupyter notebook when you run the image
# CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --ip 0.0.0.0 --no-browser --allow-root"]
CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter lab --ip 0.0.0.0 --no-browser --allow-root"]
# CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter lab --ip 0.0.0.0 --allow-root --ip=0.0.0.0 --no-browser --ServerApp.trust_xheaders=True --ServerApp.disable_check_xsrf=False --ServerApp.allow_remote_access=True --ServerApp.allow_origin='*' --ServerApp.allow_credentials=True"]

# # Use the FROM instruction to pull other images to base your new one on
# FROM nvidia/cuda:11.6.2-devel-ubuntu20.04

# # ENV PATH="/root/anaconda/bin/"

# # Install Python and pip (pip3)
# RUN apt-get -y update && apt-get install -y apt-utils python3-pip && pip3 install --upgrade pip && apt -y install wget git htop vim && rm -rf /var/lib/apt/lists/*

# # # install anaconda
# # RUN wget https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh -O condainstall.sh && \
# #     bash condainstall.sh -b -p $HOME/anaconda && eval "$(/root/anaconda/bin/conda shell.bash hook)" && \
# #     rm condainstall.sh
# #     # /root/anaconda/bin/conda init && \
# #     # echo y | /root/anaconda/bin/conda create -n po python==3.8

# # Some installs to facilitate the process.
# # RUN /root/anaconda/bin/conda init bash

# # SHELL ["/root/anaconda/bin/conda", "run", "-n", "po", "/bin/bash", "-c"]
# # RUN /root/anaconda/bin/conda install -y pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch && \
# #     pip install cvxpy cvxpylayers && \
# #     /root/anaconda/bin/conda install -y matplotlib pandas
# # RUN /root/anaconda/bin/conda install -y matplotlib

# # #  It's imperative that you install jupyter, in particular, to work with Gradient
# # RUN pip --no-cache-dir install jupyterlab
# # # RUN pip install notebook

# # # Use EXPOSE to instruct the image to expose ports as needed
# # EXPOSE 8888


# # # The main purpose of a CMD is to provide defaults for an executing container
# # # This CMD opens the jupyter notebook when you run the image
# # CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter-lab --ip 0.0.0.0 --allow-root --ip=0.0.0.0 --no-browser --ServerApp.trust_xheaders=True --ServerApp.disable_check_xsrf=False --ServerApp.allow_remote_access=True --ServerApp.allow_origin='*' --ServerApp.allow_credentials=True"]

# #  It's imperative that you install jupyter, in particular, to work with Gradient
# RUN pip install jupyter 

# # Use wget to grab files of interest to have in the container
# # RUN apt-get install -y wget

# # A sample notebook to use to confirm Tensorflow works
# RUN wget https://raw.githubusercontent.com/gradient-ai/TensorFlow/main/quick_start_beginner.ipynb


# # Use EXPOSE to instruct the image to expose ports as needed
# EXPOSE 8888


# # The main purpose of a CMD is to provide defaults for an executing container
# # This CMD opens the jupyter notebook when you run the image
# CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --ip 0.0.0.0 --no-browser --allow-root"]





# # Use the FROM instruction to pull other images to base your new one on
# FROM docker.io/paperspace/tensorflow:2.0.0-gpu-py3-jupyter-lab

# # Use the RUN instruction to make the image do a terminal command like behavior
# # Prevent conflicts with pip3
# RUN mv /usr/local/bin/pip /usr/local/bin/pip_2

# # Install Python and pip (pip3)
# RUN apt-get -y update && apt-get install -y python3-pip && pip3 install --upgrade pip

# RUN rm /usr/local/bin/pip && mv /usr/local/bin/pip_2 /usr/local/bin/pip


# # Some installs to facilitate the process.
# RUN pip install \
#     tensorflow-gpu \
#     scikit-learn \
#     scipy \
#     sklearn \
#     pandas \
#     Pillow \
#     matplotlib \
#     h5py \
#     mock \
#     certifi \
#     configparser \
#     enum34 \
#     funcsigs \
#     pathlib2 \
#     pbr \
#     scandir \
#     singledispatch \
#     webencodings 

# #  It's imperative that you install jupyter, in particular, to work with Gradient
# RUN pip install jupyter 

# # Use wget to grab files of interest to have in the container
# RUN apt-get install -y wget

# # A sample notebook to use to confirm Tensorflow works
# RUN wget https://raw.githubusercontent.com/gradient-ai/TensorFlow/main/quick_start_beginner.ipynb


# # Use EXPOSE to instruct the image to expose ports as needed
# EXPOSE 8888


# # The main purpose of a CMD is to provide defaults for an executing container
# # This CMD opens the jupyter notebook when you run the image
# CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --ip 0.0.0.0 --no-browser --allow-root"]

资源重置后自动启动

该平台6小时强制关闭实例（数据都是持久化保存的），每次手动重启太傻了，还好平台有API。

脚本功能：

  1. 固定访问链接，由于每次重启后平台会给一个随机的jupyter notebook访问链接，所以需要做成固定链接，然后做一个301跳转。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# -*- encoding: utf-8 -*-
'''
@File    :   gradient_server.py
@Time    :   2022/05/14 15:16:43
@Author  :   olixu 
@Version :   1.0
@Contact :   273601727@qq.com
@WebSite    :   https://blog.oliverxu.cn
'''

# here put the import lib
from fastapi import FastAPI
from starlette.responses import RedirectResponse

app = FastAPI()


def get_url():
    from gradient import NotebooksClient
    api_key='你的api key'
    notebooks_client = NotebooksClient(api_key)
    response = notebooks_client.list(
            tags=[]
        )
    status = response[0].state
    notebook_url = response[0].fqdn
    notebook_id = response[0].id
    token = response[0].token
    print("运行状态：", status)  # Str数据类型
    print("notebook-url链接：", notebook_url)
    print("notebook-token：", token)
    print("构造后的url链接：", notebook_url+"/lab/?token="+token)
    return "https://" + notebook_url+"/lab/?token="+token

@app.get("/oliverxu_gradient")
async def root():
    url = get_url()
    response = RedirectResponse(url=url)
    print(response)
    return response
    # return {"message": url}

脚本功能：

  1. 检测实例是否关闭，如果关闭，则开启，还有很多API，自定义玩耍的空间很大，但是我暂时只需要这些功能。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# -*- encoding: utf-8 -*-
'''
@File    :   gradient_test.py
@Time    :   2022/05/14 14:31:39
@Author  :   olixu 
@Version :   1.0
@Contact :   273601727@qq.com
@WebSite    :   https://blog.oliverxu.cn
'''

# here put the import lib
from gradient import NotebooksClient
import pdb
import time

api_key='你的api key'

notebooks_client = NotebooksClient(api_key)

while True:
    try:
        time.sleep(5)
        response = notebooks_client.list(
            tags=[]
        )
        status = response[0].state
        notebook_url = response[0].fqdn
        notebook_id = response[0].id
        token = response[0].token
        print("运行状态：", status)  # Str数据类型
        print("notebook-url链接：", notebook_url)
        print("notebook-token：", token)
        print("构造后的url链接：", notebook_url+"/lab/?token="+token)
        # pdb.set_trace()
        if status == "Cancelled":
           notebooks_client.start(id=notebook_id,machine_type='Free-GPU')
        # time.sleep(30)
        # pdb.set_trace()
    except Exception as e:
        print(e)

最终效果：

  1. 无需再登录paperspace手动点开启之类的操作（还遇到一个问题，有时候Free-GPU套餐往往会提示当前没有空闲资源，所以自动化操作每5秒钟轮询一次，爽歪歪，不会再出现资源不够的情况）
  2. 访问我的域名https://xxx.xxx/oliverxu_gradient会跳转到notebook的链接，实现固定链接访问。
If you like my blog, please donate for me.