word2vec corpus download

So you wanted to word2vec over wikipedia and tried to find ready corpus online but these aren’t that easy to find since the idea is that you run the python module and make your own.
https://radimrehurek.com/gensim/tutorial.html

Well here it is on public hosting compressed hard.
It’s not that stale it is run on whole wikipedia at around 2016-06-24

https://www.dropbox.com/s/kvihx9846irrmgy/result_wiki_en_bow.mm.index?dl=0
https://www.dropbox.com/s/pc07okw9pnh2oic/result_wiki_en_bow.mm.xz?dl=0
https://www.dropbox.com/s/21g9hm871vtvlvo/result_wiki_en_tfidf.mm?dl=0
https://www.dropbox.com/s/1oc63j6lqn4fn40/result_wiki_en_tfidf.mm.index?dl=0
https://www.dropbox.com/s/3gyy9291afyjhji/result_wiki_en_wordids.txt.bz2?dl=0
https://www.dropbox.com/s/cp3xap5xtlcq46a/result_wiki_en.tfidf_model?dl=0
https://1drv.ms/u/s!AnzVwlBMXGkvdM7biNCT_LeRsEM

Machine learning the complete guide

There is a nice wiki book composed out of wikipedia pages about machine learning
Machine Learning – the Complete Guide
the only issue with it is that you can’t download it as PDF
so alternative is to just go and download all the articles one by one
here is how to quickly do it
1. go to page open console type

$("#mw-content-text a").slice(14).map(function(i,v){return console.log(v.href);});

2. now copy paste all those links and save to file

3. run this to remove quotes from links

sed 's/\"//g' file > unquoted_links_file

4. now wget it

wget -E -H -k -K -p -i unquoted_links_file

5. profit!

ember-validations with ember-rapid-forms with backend auth

Have you ever wanted to have ember validations work with ember rapid forms all together with  errors you get back from rails?

And have you gone from blog post to blog post only to find outdated solutions?

Well this blog posts tries to do something about it.

so lets say you followed all the documentation and at present day you are left in situation where ember-validations mixin makes normal errors from ember model not work because it overwrote the errors object on the model in order to customise it.
Once you make rails return what is described in the ember error docs http://emberjs.com/api/data/classes/DS.Errors.html

you get it kinda half way so now rails part is ok and you touch nothing there.

only thing that is left it is to make validations work with native errors. We would like something seamless if possible.

so there are 3 methods in use by native ember errors that are needed for their normal operation

if we redirect them to do function on object that ember validation mixin uses then it is something that does the job.
so lets say you instantiate a model in a route that is an oportunity for patching it

model(){
  let user = this.store.createRecord('user');
  let x=user.get('errors');
  //first lets patch the add function
  x.add=function(k,v){
    //get the field on the error object that has errors for a field
    let xx = this.get(k);
    //add the error text to the array of errors for this specific field
    xx.pushObject(v[0].detail);
  }
  // if we want to remove error we just create fresh array that should hold errors for this specific field
  x.remove= function(k){this.set(k,Ember.A());}
  //once we want to clear all errors on all fields we loop trough all fields on error object and remove all messages from them
  x.clear= function(){
    let xx = this.get('errors');
    xx.forEach(function(k){x[k].unshift();});
  }
return user;
},

so error object that is created by ember validations has fields for each model field.
values of those fields are arrays with error messages
so email field can have array [’email already taken’]
so error object looks like this
{
email: [’email taken’]
}
keep in mind it is an ember object and not plain javascript it is also enumerable
now the thing should be working
 

 

motion detector

One day I was buying groceries at Lidl and while going to buy jogurt I spoted they were selling solar powered motion detector reflectors. That’s can serve as a nice asynchronous trigger for alarm system. You can just use python script to scan for motion but this is cool too.

So poped open it’s circuit box. Then mesured voltages between all the things. Then triggered the motion detector so that light turns on. Then measured the change. Then attached two wires two to two of the nodes there since input was just right for arduino or rpi. Hoocked it to arduino that signald the pi to take picture. It has rechargable solar battery.

sensor hack

Designing virtual machine

started to build small vm for fun.

Instructions coded in huffman coding with respect to this page http://www.strchr.com/x86_machine_code_statistics

It’s going to be in c++ but without fancy stuff. Keeping it low level by choice.
https://github.com/NikolaMandic/vm

microservice nectar

microservices are cyber nowdays

here are some talks that describe them
First Talk from Martin Flower tells you what are microservices and so on

Now Ms. Dehghani is lead consultant about these stuff at this company and gives very good talk on this topic
https://www.thoughtworks.com/insights/blog/microservices-lessons-frontline

Then this talk surveys how things are done now very good survey

Seems like seasoned consultant his talk delivers good points

Now this is a free tool that looks really good

run mail server on raspberry pi

have you ever wanted to run your own mail server?

But the postfix configuration made you suspect that NSA made it hard on purpose :-DDD .

Most probably that’s not the case but nowdays with some quick docker cyber we can have own mail in a snap.

Firt thing you’ll need raspberry pi. Then you’ll want to checkout docker mail project. I’ve created a fork that works on raspberry pi.

Then you get certificates from letsencrypt. And then if your provider blocks port 25 you buy a small web server and forward it’s ports like port 25 to your local pi.
how to forward:
ssh -R your_remote_server:25:ip_of_docker_container_running_on_pi:25 root@your_remote_ip

oh and pick hypriot as your os since that will run docker out of the box.

for traffic going to 25 port you can route all traffic trough tor so that it will look like you are comunicating trough tor and people who receive email will get traffic trough tor.

for that checkout https://blog.jessfraz.com/post/routing-traffic-through-tor-docker-container/

Alternative to tor variant is that you use ssh tunnel to route traffic back and forth from your host

add this to sshd conf
PermitTunnel yes

so you would open two tunnel interfaces to connect your remote to local pi

ssh -f -w 0:1 root@yourremote true

then give ip to your local
ifconfig tun0 10.1.1.1 10.1.1.2 netmask 255.255.255.252

give ip to your remote interface on your remote computer
ifconfig tun1 10.1.1.2 10.1.1.1 netmask 255.255.255.252

transef traffic from remote tunnel to remote public interface eth0
iptables -I FORWARD -i tun1 -o eth0 -j ACCEPT
then make address translation on that remote interface so that outside world sees your remote computers ip
iptables -A POSTROUTING -t nat -o tun1 -p tcp –dport 25 -j SNAT –to yourremoteip
turn on forwarding
sysctl -w net.ipv4.conf.all.forwarding=1
you need to create this route in remote routing table
10.1.1.0/30 dev tun1 proto kernel scope link src 10.1.1.1

on local
create new routing table by adding this line
200 mail-route
in
/etc/iproute2/rt_tables
then set that default gateway for that routing table is the local tunnel interface
ip route add default via 10.1.1.2 dev tun0 table mail-route

now mark all traffic to port 25 with mark 1
iptables -A OUTPUT -t mangle -o tun0 -p tcp –dport 25 -j MARK –set-mark 1

now all with mark 1 will use our new routing table and will all go trough tunnel
ip rule add fwmark 0x1 table mail-route

so not sure if I wrote down all the steps but the idea is

you create ssh tunnel, give ips to those devices and then create snat on exit now you need to redirect traffic from docker to local tunnel interface

this is the iptables on local that enables connecting to smtp port 25 over remote computer when outband 25 is blocked by provider

Chain PREROUTING (policy ACCEPT 28 packets, 1802 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 2 packets, 120 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        9   540 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 1 packets, 60 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       29  1862 SNAT       all  --  *      tun0    0.0.0.0/0            0.0.0.0/0            to:172.17.0.2
2        1    60 SNAT       all  --  *      tun0    0.0.0.0/0            0.0.0.0/0            to:10.1.1.2
3       23  1652 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
4        0     0 MASQUERADE  tcp  --  *      *       172.17.0.2           172.17.0.2           tcp dpt:993
5        0     0 MASQUERADE  tcp  --  *      *       172.17.0.2           172.17.0.2           tcp dpt:587
6        0     0 MASQUERADE  tcp  --  *      *       172.17.0.2           172.17.0.2           tcp dpt:143
7        0     0 MASQUERADE  tcp  --  *      *       172.17.0.2           172.17.0.2           tcp dpt:25

Chain DOCKER (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:993 to:172.17.0.2:993
2        0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:587 to:172.17.0.2:587
3        0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:143 to:172.17.0.2:143
4        0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25 to:172.17.0.2:25

for this to work you need
iptables -I POSTROUTING -t nat -o tun0 -j SNAT –to 10.1.1.2
this still does not make docker work
in order to make it work for docker another route needs to be added
this is for the traffic comming back to internal tun interface
so that once it comes back it knows where is the docker so it can send it to the container

ip route add 172.17.0.0/16 via 172.17.0.1 dev docker0 table mail-route
ip route add 192.168.1.0/24 via 192.168.1.35 dev eth0 table mail-route

so here is the complete system config dump

file that says what routing tables do we have

$ cat /etc/iproute2/rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
200 mail-route
#
# local
#
#1      inr.ruhep

ip tables on local

{ ~ }  » cat local.fw                                                                                                                                                                                                                    ~
# Generated by iptables-save v1.4.21 on Mon Dec 28 23:59:44 2015
*mangle
:PREROUTING ACCEPT [19476:1370924]
:INPUT ACCEPT [16670:1217347]
:FORWARD ACCEPT [1412:95083]
:OUTPUT ACCEPT [18094:8090507]
:POSTROUTING ACCEPT [19723:8192534]
-A OUTPUT -o docker0 -p tcp -m tcp --dport 25 -j MARK --set-xmark 0x1/0xffffffff
-A OUTPUT -o tun0 -p tcp -m tcp --dport 25 -j MARK --set-xmark 0x1/0xffffffff
-A OUTPUT -o eth0 -p tcp -m tcp --dport 25 -j MARK --set-xmark 0x1/0xffffffff
-A OUTPUT -o eth0 -p tcp -m tcp --dport 25 -j MARK --set-xmark 0x1/0xffffffff
-A OUTPUT -o docker0 -p tcp -m tcp --dport 25 -j MARK --set-xmark 0x1/0xffffffff
-A OUTPUT -o docker0 -p tcp -m tcp --dport 25 -j MARK --set-xmark 0x1/0xffffffff
-A POSTROUTING -o docker0 -p tcp -m tcp --dport 25 -j MARK --set-xmark 0x1/0xffffffff
COMMIT
# Completed on Mon Dec 28 23:59:44 2015
# Generated by iptables-save v1.4.21 on Mon Dec 28 23:59:44 2015
*nat
:PREROUTING ACCEPT [2186:141299]
:INPUT ACCEPT [462:63786]
:OUTPUT ACCEPT [887:62538]
:POSTROUTING ACCEPT [892:62838]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -o tun0 -p tcp -m tcp --dport 25 -j SNAT --to-source 10.1.1.2
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 993 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 587 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 143 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 25 -j MASQUERADE
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 993 -j DNAT --to-destination 172.17.0.2:993
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 587 -j DNAT --to-destination 172.17.0.2:587
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 143 -j DNAT --to-destination 172.17.0.2:143
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 25 -j DNAT --to-destination 172.17.0.2:25
COMMIT
# Completed on Mon Dec 28 23:59:44 2015
# Generated by iptables-save v1.4.21 on Mon Dec 28 23:59:44 2015
*filter
:INPUT ACCEPT [16389:1201691]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [17774:7850871]
:DOCKER - [0:0]
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 993 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 587 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 143 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 25 -j ACCEPT
COMMIT
# Completed on Mon Dec 28 23:59:44 2015

-A FORWARD -i tun1 -o eth0 -j ACCEPT
-A POSTROUTING -o tun1 -p tcp -m tcp --dport 25 -j SNAT --to-source 10.1.1.1
-A POSTROUTING -o tun1 -p tcp -m tcp --dport 25 -j SNAT --to-source 46.4.83.143

dont forget to turn on forwarding with sysctl command
also run tcp dump for debugging etc.
 tcpdump -vvv -i tun0  
 tcpdump -vvvv -i tun1
tcpdump -vvvv -i eth0  dst host 10.1.1.1
tcpdump -vvvv -i eth0  dst host 10.1.1.1


                    +-------------+                                                  raspb pi
                    |             |
                    |             |                                             +--------------------+
                    |             |        port going to 25   over tunnel       |                    |
                    | +---------+ |                                             | +---------+        |
                    | |snat tun1| |    <----------------------------------+     | |  tun0   |        |
 contact servers    | |         | |                                             | |         |        |
 over remote        | +---------+ |    +---------------------------------->     | +---------+        |
                    |     |   ^   |                                             |                    |
  +------------->   | +---v-----+ |                                             | +----------------+ |
                    | |snat eth0| |     all things going over eth0 for port 25  | |  container     | |
   <--------------+ | |         | |     go into tun0 instead     +------>       | |  mail          | |
                    | +---------+ |                                             | |                | |
                    |             |                                             | |                | |
                    |             |                                             | +----------------+ |
                    |             |                                             |                    |
                    |             |                                             |    ^   +  ^   +    |
                    |             |                                             |    |   |  |   |    |
                    |   +-----+   |  <-----------------25------------------->   |    |   |  |   |    |
         port forw. |    |-----+  |  <-----------------143|----------------->   |    |   |  |   |    |
                    |    |-----+  |  <----------------993------------------->   |    |   |  |   |    |
                    |   +-----+   |  <---------------587-------------------->   |    +   v  +   v    |
                    |             |                                             |                    |
                    +-------------+                                             +--------------------+



docker from scratch

people seems to be using docker images that are just packed binaries.
With some yak shawing it should be possible to do erlang on docker from scratch. 😀 but not gonna yak shave today