Обновить

Single exit point to web, I2P, TOR and blocking bypass

Время на прочтение 6 min
Количество просмотров 23K

A sly plan


Preamble... This article was written back in the summer but, for reasons beyond the author’s control, was a little delayed...


One day, on a hot summer evening, after another entry into the console browser commands like :set content.proxy socks://localhost:9050, The author of this opus realized that it is impossible to live like this any longer and it’s time to bring access to all sorts of hidden networks, and at the same time bypassing blocking of the name of a well-known organization, to some single “common denominator” for any software in general and browser in particular. How to bring it? Of course, so that the proxy server itself “understands” through which higher proxy to send and receive traffic depending on the entered address. The second goal, which follows from the first, is that upstream proxies can operate as either http or socks and both protocols must be supported by the ingress proxy. Well, the software itself should be more or less up-to-date, so that in case of errors or “wanting features”, you don’t have to sadly look at a lonely turnip on Github, or even on some Sorsforge.
So the goals are set!


The agony of choice


In fact, there was no particular pain. Because, by and large, of the available known proxy servers, two met the requirements. This privoxy And tinyproxy. But tinyproxy turned out to be more lively, more lightweight and simpler, so it was chosen and immediately installed (using the current version of Manjaro Linux as an example).


sudo pacman -Syu tinyproxy

Of course, before this, the packages were already installed tor And i2pd. (sudo pacman -Syu tor i2pd).


Basic tinyproxy setup


So, let’s set up basic redirections so that the regular web goes directly, and *.i2p And *.onion through the corresponding parent proxies.


/etc/tinyproxy/tinyproxy.conf:


User tinyproxy
Group tinyproxy
PidFile "/var/run/tinyproxy/tinyproxy.pid"

Port 8888
Listen 127.0.0.1
Timeout 600

DefaultErrorFile "/usr/share/tinyproxy/default.html"
StatFile "/usr/share/tinyproxy/stats.html"

Syslog On
# Set the logging level. Allowed settings are:
#   Critical, Error, Warning, Notice, Connect, Info
LogLevel Info

MaxClients 100
MinSpareServers 5
MaxSpareServers 20
StartServers 10
MaxRequestsPerChild 0

Allow 127.0.0.1

ViaProxyName "tinyproxy"

## Parent proxy for TOR hosts
upstream socks5 127.0.0.1:9050 ".onion"
## Parent proxy for I2P hosts
upstream socks5 127.0.0.1:4447 ".i2p"

##### End of static configuration #####

To begin with, almost all parameters in the config remain default.


  • Save
  • Let's launch: sudo systemctl enable --now tinyproxy
  • Checking: journalctl -f -u tinyproxy, We are simultaneously trying to access i2p and onion resources (by setting the browser to use http proxy http://localhost:8888) and see in the log file messages about redirections to the parent proxy's:
    июл 20 01:36:16 dell-lnx tinyproxy[198356]: Request (file descriptor 6): GET http://flibusta.i2p/ HTTP/1.1
    июл 20 01:36:17 dell-lnx tinyproxy[198356]: Found upstream proxy socks5 127.0.0.1:4447 for flibusta.i2p
    июл 20 01:36:17 dell-lnx tinyproxy[198356]: opensock: opening connection to 127.0.0.1:4447
    июл 20 01:36:17 dell-lnx tinyproxy[198356]: opensock: getaddrinfo returned for 127.0.0.1:4447
    июл 20 01:36:17 dell-lnx tinyproxy[198356]: Established connection to socks5 proxy "127.0.0.1" using file descriptor 7.
    июл 20 01:36:40 dell-lnx tinyproxy[198356]: Closed connection between local client (fd:6) and remote client (fd:7)
    ...
    июл 20 01:39:36 dell-lnx tinyproxy[214495]: Found upstream proxy socks5 127.0.0.1:9050 for ilitafrzzgxymv6umx2ux7kbz3imyeko6cnqkvy4nisjjj4qpqkrptid.onion
    июл 20 01:39:36 dell-lnx tinyproxy[214495]: opensock: opening connection to 127.0.0.1:9050
    июл 20 01:39:36 dell-lnx tinyproxy[214495]: opensock: getaddrinfo returned for 127.0.0.1:9050
    июл 20 01:39:36 dell-lnx tinyproxy[214495]: Established connection to socks5 proxy "127.0.0.1" using file descriptor 7.

List of "banned.info"»


Well, the proxy combination works basically, now the fun begins - bypassing Roskomnadzor blocking. Unfortunately, tinyproxy does not support external files with a parent proxy list, but this should not be a barrier. After all, we can generate a monolithic config “on the fly”».


  1. Copy the existing tinyproxy config under a new name:


    cp /etc/tinyproxy/tinyproxy.conf /etc/tinyproxy/tinyproxy.conf.static

  2. We slightly edit the new one /etc/tinyproxy/tinyproxy.conf.static: LogLevel InfoLogLevel Error


  3. Create a unit that will clone the project repository ban.infosudo systemctl edit --full --force z-i-prepare.service:


    # /etc/systemd/system/z-i-prepare.service
    [Unit]
    Description=Zapret Info repository cloner
    ConditionPathExists=|!/usr/local/lib/z-i/
    ConditionFileNotEmpty=|!/usr/local/lib/z-i/dump.csv
    Wants=local-fs.target
    After=local-fs.target
    Wants=network.target
    After=network.target
    #
    [Service]
    Type=oneshot
    User=tinyproxy
    Group=tinyproxy
    ExecStartPre=+/usr/bin/mkdir -p /usr/local/lib/z-i
    ExecStartPre=+/usr/bin/chown tinyproxy:tinyproxy /usr/local/lib/z-i
    ExecStartPre=+/usr/bin/chmod 0750 /usr/local/lib/z-i
    ExecStart=git clone --depth 1 https://github.com/zapret-info/z-i.git /usr/local/lib/z-i

  4. We create a unit that will generate the tinyproxy config at runtime — sudo systemctl edit --full --force tinyproxy-cfg-generator.service:


    # /etc/systemd/system/tinyproxy-cfg-generator.service
    [Unit]
    After=z-i-prepare.service
    Wants=z-i-prepare.service
    #
    [Service]
    Type=oneshot
    User=tinyproxy
    Group=tinyproxy
    Environment="PATH=/usr/local/bin:/usr/sbin:/usr/bin"
    ExecStart=tinyproxy-cfg-gen.sh
    StandardOutput=file:/run/tinyproxy/tinyproxy.conf

    … and the script itself /usr/local/bin/tinyproxy-cfg-gen.sh to him:


    #!/usr/bin/env sh
    # tinyproxy-cfg-gen.sh -- tinyproxy dynamic config generator to stdout.
    awk -F';' '{print "upstream socks5 127.0.0.1:9050 \"" $2"\""}' /usr/local/lib/z-i/dump.csv|tr -d '*'|sort|uniq|grep -v '\s\"\"'>/tmp/tinyproxy.conf.dynamic
    cat /etc/tinyproxy/tinyproxy.conf.static /tmp/tinyproxy.conf.dynamic

  5. Timer and service that will pump out list updates once a day and restart the main service: sudo systemctl edit --full --force z-i-update-daily.timer:


    # /etc/systemd/system/z-i-update-daily.timer
    [Unit]
    Description=Zapret Info daily update
    After=network.target
    Wants=network.target
    #
    [Timer]
    OnCalendar=daily
    AccuracySec=1h
    Persistent=true
    #
    [Install]
    WantedBy=timers.target

    And the service for it sudo systemctl edit --full --force z-i-update-daily.service:


    # /etc/systemd/system/z-i-update-daily.service
    [Unit]
    Description=Zapret Info daily update service
    After=network.target
    Wants=network.target
    #
    [Service]
    Type=oneshot
    User=tinyproxy
    Group=tinyproxy
    ExecStartPre=/usr/bin/git -C /usr/local/lib/z-i pull
    ExecStart=+/usr/bin/systemctl try-restart tinyproxy.service

  6. Finally, the cherry on the cake, we edit tinyproxy.service to suit our needs — sudo systemctl edit tinyproxy.service:


    # /etc/systemd/system/tinyproxy.service.d/override.conf
    [Unit]
    Wants=network.target
    Wants=z-i-prepare.service
    After=z-i-prepare.service
    Wants=tinyproxy-cfg-generator.service
    After=tinyproxy-cfg-generator.service
    #
    [Service]
    User=tinyproxy
    Group=tinyproxy
    ExecStart=
    ExecStart=/usr/bin/tinyproxy -c /run/tinyproxy/tinyproxy.conf
    ExecStopPost=+/usr/bin/rm -rf /run/tinyproxy/tinyproxy.conf

  7. And now, with all this disgrace, we will try to take off ©


    sudo systemctl enable --now tinyproxy
    sudo systemctl enable --now z-i-update-daily.timer

    How it works?



A thoughtful reader will certainly be interested in, what are these dances with a tambourine for? Well, in conclusion it doesn’t hurt to clarify some points. Let's go straight to the points of the previous section.


  1. Everything is simple here. We save in a separate file that part of the configuration that should not change automatically.
  2. A very important parameter that reduces the loading time of the main service from an hour (SIC!) to about a minute and a half (netbook AMD processor from 2009 and HDD at 5400 rpm). Of course, this is not the only way to improve productivity..
  3. «bootstrap" unit that always runs, but only runs if the directory is missing /usr/local/lib/z-i/ or the file is empty /usr/local/lib/z-i/dump.csv (Directives Condition*). Key --depth 1 allows you to clone only the last commit, not the entire 8 GB.
  4. The main generation of the config and another live hack to improve performance. From csv
    awk-Then the field with the domain is cut out and cleared of unnecessary characters. Rows with an empty domain are deleted, then cat sends the result to stdout. and already a unit, thanks to the directive StandardOutput= writes all output to a config file in /run on tmpfs! According to dependencies, it starts after the “bootstrap” unit from the previous paragraph has completed.
  5. Once a day, starting from zero hours, with jitter per hour, we update the repository and regenerate the config, with a service restart. More precisely, we restart the service with regeneration of the config.
  6. (and 7.) Well, everything is clear here, launching auxiliary units and the main.

This link is already working a week 2,5 month. Glucose bugs do not seem to have been noticed yet. ready-made configs and scripts live on github, PR are welcome!

Tags:
Hubs:
Всего голосов 62: ↑61 и ↓1 +60
Комментарии 63
+63

Comments 63

Do you really go to all these prohibited sites? If not, then you can select only the ones you need, and then a proxy with 5-10 sites will start much faster, and setting up in general will be easier.

The problem is that you never know when you will stumble upon a blocked resource. Look, deviant-art was recently blocked... And even for a minute or a minute and a half, once a day, this is nonsense. But on a modern system, it’s actually seconds.

For some time now, RKN began to practice blocking unwanted traffic using so-called TSPU, without adding entries to the lists sent to providers.

Thus, in order to be a little less than guaranteed not to stumble upon a blocked resource, you need to use “white” lists: go around all resources except those indicated.

Well, so far there are a minimum of such cases. And there is either a donkey or a padishah... ©

great article, I wish I could give it a plus!

you need to use “white” lists: go around all resources except those indicated.

If it's not a secret, please share your white list. I want to switch my proxy to the specified strategy and know in advance where it is better to leave the white list. Apart from maybe the music, no ideas about content come from the blue site.

And just to make do with one comment - many thanks to the author!

I hope that based on the posted instructions, a guide/script will appear for routers so that i2p/onion/proxy and all that will be available from all wireless network devices. Of course, there are several similar guides, but everything is a little different.

It's no secret, but I don't think it's my the list will be of great use. Here he is:

mail.ru

yandex.net

yandex.ru

You can search for something "official", but I still have enough of manually listing resources that are inconvenient to access through a proxy.

PS: here-too-ru)

What about government services?!

Government services, banks, tax service, some Netflix or ivi, VKontakte

The solution is interesting, but if there is no task of uploading it completely to the darknet, why not get by with vpn through a foreign vps ?

For cases when there is no possibility. Well, even in the case of VPS, if you want to drive only traffic to blocked resources, on VPS instead of TOR, install the same tinyproxy and register it on this side.

yes, but in the case of vps it can be done without a proxy, only by routing and tunneling. This allows you to raise the tunnel at the level of your home router and all devices connecting to the router will transparently travel through foreign countries. The advantage of this solution is that there is no need to install anything third-party on clients, and there is no need to configure proxies in applications. If you make a vpn on ipsec, then you can use the same vps to encrypt traffic when using public wi-fi with any iron.

Any tunneling is, one way or another, an overhead and a loss of performance. For example, why do I need to download OS updates from Manjaro via VPS/VPN? And so you have to drive DNS traffic in roundabout ways, because the provider replaces the responses from the “four stakes” and other public DNS, in case of an attempt to resolve something that is prohibited. But, as it seems to me, most of the traffic may well go directly.

Fair point. However, it is worth noting that using the mangle branch you can mark which traffic is sent to the tunnel and which is not. Of course, there is no point in arguing about the overhead, it exists, but how noticeable it is is a big question. I’ve been living with the previously described scheme for several years, I’m looking at what other options there are to bypass the blocking... thanks for the article.

If you tried V2ray via quic it would be very interesting to read about your experience :)

I haven't tried it. Can I have a link??

I'm not good at these technologies. But as far as I understand, tor is something similar to a torrent (not only with the same three letters), that is, through my computer there will be traffic that may not be very good, not only from the point of view of the RKN, but from any point of view. Or I'm wrong?

TOR can work simply as a client (by default), like a relay - transmitting someone else's Encrypted traffic and send it further. And how Exit Node is the actual node for accessing the Internet. In the case of relay, you will never know the content of the traffic, the initial and final destinations. Because only the exit node sees the decrypted traffic (but it also does not know who it is intended for inside TOR). I repeat, TOR is configured by default as just a client. But Rayleigh does not carry any risks. In the Russian Federation, only the output node is scary. There were precedents.

There was a case with a rayleigh - they made a “purchase” and saw the address of the rayleigh. The owner of the relay had to explain that it contained a network node and did not know what kind of traffic was passing through. There were no charges after that.

Well, in general, it’s better not to shine in the Russian Federation... Although I don’t remember, it seems in the settings there was an option to prohibit being an input relay.

Do you have any ideas on what to do when RKN starts slowing down YouTube? And unfortunately he has the necessary technology for this...

VPN. And if they block at DPI, then it’s an obfuscated VPN.

This is how I check zi updates without downloading the entire repository:

cd /usr/share/zapret/z-i
if [ `git log --pretty=%H ...refs/heads/master^` == `git ls-remote origin -h refs/heads/master |cut -f1` ]; then
  	 exit 1;
fi
echo `date` - "New z-i updates"
git pull

Thank you, it will be useful!

Try `eatmydata git clone`, it might speed things up.

Yes, it downloads quickly. Most of the time is spent on “sucking in” the resulting config by tinyproxy itself.

And publish a turnip with scripts. And grab the stars and get PR to support other varieties of Linux.

So at the very end of the article, where...

This link is already working a week 2,5 month. Glucose bugs do not seem to have been noticed yet. ready-made configs and scripts live on github, PR are welcome!

And link.

Please translate the article for stupid Windows users (me and others like me). Thank you.

Oh... Was it really difficult? What exactly is unclear?

You probably need links to download tor, i2pd and tinyproxy, instructions for setting up the "Task Scheduler" and a script on PS. Or how to run it in WSL.

100500 I haven’t dealt with Windows for years, because it’s unnecessary...

On Windows I use GoodbyeDPI. As far as I understand, packets are cut in a certain way, which allows you to bypass blocking.

Yes. There, packets are fragmented, which causes many DPI systems to go a little crazy. They don't know how to collect fragmented packets.

Yeah, I used it before - it was convenient. No proxies are needed. Now it doesn’t work, there’s simply no connection with it at all. Provider MGTS

I understand correctly that this thing will not restore access to resources that are not blocked, but have a common IP address with blocked ones?

Yes... But in what comes from zapret.info csv The file also has a column with an IP address... but not in all entries. In general, I'm thinking about this question. How to make sure that the config does not swell to obscene sizes?.

Well, you can make a DNS proxy that would check all returned IP addresses to see if they are blocked. If the response contains at least one unblocked address, then we simply exclude from the response everything that is blocked; if all addresses are blocked, we replace them with some 127.0.0.2. Well, there should be a transparent proxy on 127.0.0.2.

I wonder which of our popular DNSs can do such tricks? Somehow it seems to me that there is no such thing... But this is not certain.

From the box? Nobody, using scripts to add such functionality to them is real.

Well, that's interesting. Which server can have hooks for requests/responses?.

I have pfsense at home with a tunnel to the Amsterdam vps. It’s just that everything that comes from the home network and falls under the list goes into the tunnel. Transparent for clients.

So maybe an article? More ways, good and different!

It's lazy. And the scheme is not particularly beautiful and complete. I add sites to the list myself that need to be redirected to the tunnel.
Maybe one day…

A plugin for browsers to commit a blocked domain to GitHub. I pressed the button, the domain is sent to the list and the call to the script updating the list on the proxy from Git is immediately triggered.

Another option is PDNSD. A very easy to configure caching DNS server. Maybe he can DoH? Then the response from the configured upstream DNS will not be replaced so easily.

Well, at least the basic principles. And then you see, the community will help. What are we, aren’t we automators? ;-P

A naive question: is it possible to configure a proxy so that any connection is first established directly, and if a blockage is detected, bypass mechanisms are used? The frigate lite extension for chrome once worked in a similar way, but as far as I know it is no longer available.

Hmm... Well, this will be slower than looking at the list → directing it to the right place. And then, blocking detection is not so easy to reliably catch. Especially when it comes to SSL/TLS connections.

I agree about the speed, but you can cache the state, so if a blockage is detected, then for the next day (for example) we access the resource through workarounds. As for the guaranteed detection of blocking - yes, there may be problems.

But with this approach there is no need to constantly update the config.

I don’t have many sites blocked (linkedin), some sites with torrents. In this case, I simply turn on the vpn hoxx add-on in Firefox, the free version. No settings, everything is as simple as possible. And so I have nothing to hide.

Well, I don't have Firefox. No, of course, in qutebrowser you can expand the functionality with Python scripts, but I’m still a Pythonist. Yes, and universal. Works even with torrent and IRC client.

Hello,

Wouldn't it be a better option to additionally create a .path service that will restart only if the hosts file has changed??

Thought! Thank you! Only then will it be necessary to check the result of the work git, which pulls out lists. And only if new data has arrived, regenerate the config.

And in my version, this is true, if there were no changes in the repository, the script will end with error code 1. If there were changes, the changes will be pulled through git pull and after that you can regenerate the list of hosts. You can wrap a script around it, check the return code, and make a decision. Or you can add your code immediately after git pull. It was more convenient for me to make the strapping.

I'll have to work on it this weekend...

A light solution to the problem is the FoxyProxy extension for Firefox. It allows you to set up different proxies quite flexibly for different sites. Works in windows/linux/android.

I used it before, until I left FF / Chrome. Only one problem. You also need to somehow load the list of rocket launchers into it, and something tells me that in automatic mode this will be a little more problematic.

Proxy autoconfiguratiion (PAC) allows you to write a JS script and add all the logic there. And most importantly, browsers already support it.

I installed tor locally and prepared proxy.pac, if the domain is on the block list, then tor is used, otherwise connect directly.

Proxy.pac may be on a different host.

This decision is slowing down. Either because of the number of domains or because of the file size.

Well, I didn’t add the entire list of domains blocked by RKN, just what I use myself.

"Tor itself slows down the most, but this is expected.

Well, I just used the PAC script with a complete list of domains before. It slows down more than tor.

There are a couple of problems - make the script executable (one), add the -s parameter to grep in the script (two), uncomment the PID file in the tinyproxy config (three). Well yes, a couple is about three.

Oh, and we also had to start the Tor service.

And that's how it works, yes.

So maybe an issue or PR, on Github?

Posted an email.

Only full-fledged users can leave comments. Sign in, Please.