Posts on yshalsager

Using Parsel instead of Beautiful Soup for Web Scraping

Mon, 28 Nov 2022 00:00:00 +0000

Web scraping is an automated process to extract data from web page, and since Python is one the most popular programming languages it’s common to see people use it for doing web scraping tasks like me :)

For a long time, I have been using Beautiful Soup 4 to extract data from web pages’ HTML markup, it’s popular, easy, robust, and battle-tested library for navigating, searching, and modifying the DOM tree. But, recently I came across Parsel, another HTML parsing library that supports XPath selectors, which is missing in Beautiful Soup, and I was in need of using something that can extract data from HTML using XPath (rather than Scrapy, funny enough, later I knew that Scrapy uses Parsel under the hood :D), so I decided to get it a try.

Thoughts and Tricks After Usage

Parsel is so powerful! it saved me much time than bs4 usually did, this is mainly because of the easy way it provides to access DOM Node HTML, text, attributes and other values easily with a handy way to get a default value if the required data didn’t exist.
Parsel uses lxml for parsing the web page, this can result in a huge performance improvement according to selectolax’s benchmark (which is an interesting library to try as well).
Parsel is easy to use, all you need to do is to import Selector from parsel package then use Selector(text=response.text) to load HTML string into a selector object.

1# code snippet from https://parsel.readthedocs.io/en/latest/usage.html 
2from parsel import Selector
3text = "<html><body><h1>Hello, Parsel!</h1></body></html>"
4selector = Selector(text=text)

.getall() and .get() are equivalent for .select()and .select_one() in bs4.
.xpath() and .css() methods can be chained! selector.css('img').xpath('@src').getall()
Selector’s non-standard ::text pseudo-element can be better than Beautiful Soup’s .text because it won’t be None even with empty text. Same goes for ::attr vs .get() (without default value).
Use *::text to selects all descendant text nodes of the current selector.
If you want to use regular expressions (regex) to get data from a string of a selector, all you need to do is to use .re() and .re_first() methods. However, unlike using .xpath() or .css() methods, regex methods returns a list of strings so they can’t be chained.
The proper way to work with relative XPaths is to prefix the path with dot divs.xpath('.//p').
drop() can be used to remove elements based on a Selector, this is similar to Tag.decompose() in bs4, and can’t be undone.
When querying by class, consider using CSS instead of XPath.
You can convert CSS to XPath using Parsel’s css2xpath function.
Because of lxml, sometimes you may get something rather that what browser show while parsing the HTML using Parsel, here are more details about this issue.

Extra Resources

CSS cheatsheet that has good summary of CSS selectors.
Xpath cheatsheet.
XPath Playground to try out your XPath queries.
XPath/CSS Equivalents.

Debugging and fixing Selenium's send_keys() wrong text input

Mon, 28 Sep 2020 00:00:00 +0000

There are many kinds of bugs you may face while programming, but without a doubt, ghost bugs are the worse!

I recently faced a weird problem while working on a freelance Selenium (A portable framework for testing web applications) project in Python which is: send_keys method is sending random wrong input. Here are the details of the problem, how did I debug it, and how I managed to fix after hours of investigating!

I have been working on a freelance Selenium web automation and scraping project and we need to move the script to a better Linux server that runs Ubuntu 20.04. After migrating I found out that Selenium text input became so random (although it works perfectly on local PCs). For example, spaces are no longer sent, numbers are mixed, some characters are dropped, and so forth.

I started to debug with Python’s logging standard module, but the log said that the correct input was being entered, which is logical since no code changes happened. Then, I tried to see if there’s something wrong with chromedriver that being used to automate Google Chrome browser, but again, version on the old and new servers was the same. I have also tried on another (third) server that runs Ubuntu 18.04 and the same bug occurred. It’s Google and stackoverflow time! I searched our beloved websites to see if any people faced this problem besides me. I found many results from 6 years ago till 5 months ago. The most interesting one was a chromedriver’s bug that got closed with WontFix status. This bug aside, the suggested solutions were:

Using clear() method of the WebElement before sending the text input.
Clicking the text box before sending keys.
Using WebDriver’s execute_script() method to execute JavaScript code that modifies the value of the input form to the required text.
Using sleep from time standard module to wait for seconds between two send_keys() calls.
Changing the server’s keyboard layout to US.
Using pyperclip module to paste the text into the browser directly.

But surprisingly, none of the mentioned solutions above did it, so I gave up and planned to switch to Firefox’s geckodriver in the next day.

After starting to add geckodriver support, I decided to search for a solution again since the code works perfectly on the PC. While reading stackoverflow’s solutions my eyes caught a very interesting answer:

I also experienced this problem when connecting to a VM using VNC and then running the Selenium test that way. The VNC was actually the one dropping characters. Once I moved to a direct connection to the VM using the VirtualBox console… it worked fine.

This sounded very logical to me, so I quickly check what is the used VNC on the new server (Ubuntu 20.04) and the third server (Ubuntu 18.04). It was TightVNC. The old server VNC was available externally from the service provider so I couldn’t check what does it use.

After more quick googling I found out that TigerVNC should be a good alternative to the current VNC server. Using apt (a package manager for Debian, Ubuntu, and related Linux distributions), I installed tigervnc-standalone-server tigervnc-common tigervnc-xorg-extension packages and boom! it worked like a charm.

It was a very weird problem that occurred out of the blue and I took a very long time to figure it out and fix it. Hopes this post helps you if came here for fixing this problem!

How to build a Project Treble GSI ROM from source?

Sat, 09 Jun 2018 00:00:00 +0000

Note: This guide is super old and some steps might be changed, please refer to the community edited guide for a more recent guide.

As you read this guide now I’ll assume you already have a previous knowledge about How to build android from source, so I won’t cover some points with too many basic details. So, let’s start:

What you’ll need:

A treble enabled device, basically all devices that come with Android 8.1 out of box support it.
A relatively recent 64-bit computer (Linux, OS X, or Windows) with a reasonable amount of RAM and about 100 GB of free storage (more if you enable ccache or build for multiple devices). The less RAM you have, the longer the build will take (aim for 8 GB or more). Using SSDs results in considerably faster build times than traditional hard drives.
A USB cable compatible with your device
A decent internet connection & reliable electricity
Some familiarity with basic Android operation and terminology. It would help if you’ve installed custom ROMs on other devices and are familiar with recovery. It may also be useful to know some basic command line concepts such as cd for “change directory”, the concept of directory hierarchies, that in Linux they are separated by /. etc.

Summary

Installing SDK
Installing build packages
Install the repo command
Configuring git
Turning on caching to speed up build
Building using phhusson’s script
Building using dakkar’s script
Building using the manual way

1. Installing SDK

If you haven’t previously installed adb and fastboot, you can download them from Google. Extract it using:

1unzip platform-tools-latest-linux.zip -d ~

Now we have to add adb and fastboot to our path. Open ~/.profile and add the following:

1# add Android SDK platform tools to path
2if [ -d "$HOME/platform-tools" ] ; then
3 PATH="$HOME/platform-tools:$PATH"
4fi

Then, run this to update your environment.

1source ~/.profile

2. Installing build packages

Several packages are needed to build Android. You can install these using your distribution’s package manager. You’ll need:

1bc bison build-essential curl flex g++-multilib gcc-multilib git gnupg gperf imagemagick lib32ncurses5-dev
2lib32readline-gplv2-dev lib32z1-dev libesd0-dev liblz4-tool libncurses5-dev libsdl1.2-dev libwxgtk3.0-dev
3libxml2 libxml2-utils lzop pngcrush schedtool squashfs-tools xsltproc zip zlib1g-dev openjdk-8-jdk

3. Installing the repo command

Enter the following to download the repo binary and make it executable (runnable):

1mkdir -p ~/bin
2curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
3chmod a+x ~/bin/repo

Put the ~/bin directory in your path of execution

In recent versions of Ubuntu, ~/bin should already be in your PATH. You can check this by opening ~/.profile with a text editor and verifying the following code exists (add it if it is missing):

1# set PATH so it includes user's private bin if it exists
2if [ -d "$HOME/bin" ] ; then
3 PATH="$HOME/bin:$PATH"
4fi

Then, use this to update your environment.

1source ~/.profile

4. Configuring git

You’ll need to set up git identity in order to sync the source, run these commands:

1git config --global user.name "your username"
2git config --global user.email yourmail@example.com

5. Turning on caching to speed up build

You can speed up subsequent builds by running:

1export USE_CCACHE=1
2export CCACHE_COMPRESS=1

And adding that line to your ~/.bashrc file. Then, specify the maximum amount of disk space you want the cache to use by typing this from the top of your Android tree:

1prebuilts/misc/linux-x86/ccache/ccache -M 50G

Where 50G corresponds to 50GB of cache. This needs to be run once. Anywhere from 25GB-100GB will result in very noticeably increased build speeds (for instance, a typical 1hr build time can be reduced to 20min). If you’re only building for one device, 25GB-50GB is fine. If you plan to build for several devices that do not share the same kernel source, aim for 75GB-100GB. This space will be permanently occupied on your drive, so take this into consideration. See more information about ccache on Google’s Android build environment initialization page.

6. Building using phhusson’s script

We can’t deny that @phhusson has made amazing works and countless contributions to Project Treble ROMs development apart from his experimentations is a build script which make build a GSI super simple job. By default, it supports building LineageOS - RR - Carbon. 1- Open your terminal and run:

1git clone https://github.com/phhusson/treble_experimentations

2- To clone and build enter the following command and replace “romname” with lineage|rr|carbon

1mkdir romname; cd romname
2bash ../treble_experimentations/build-rom.sh android-8.1 romname

3- The script will automatically initialize the repository, sync the source, apply patches and start building.

7. Building using dakkar’s script

dakkar’s script is another treble building script, originally made by @Dakkar and improved by contributors on treble experimentations repo. It’s customizable, easy to understand and can build almost all ROMs with simple edits.

1- Open your terminal and run:

1git clone https://github.com/phhusson/treble_experimentations

2- To start the process:

1bash ../treble_experimentations/build-dakkar.sh romname variant

Variants are dash-joined combinations of (in order):

processor type

“arm” for ARM 32 bit

“arm64” for ARM 64 bit

A or A/B partition layout (“aonly” or “ab”)

GApps selection

“vanilla” to not include GApps

“gapps” to include opengapps

“go” to include gapps go

SU selection (“su” or “nosu”) for example:

arm-aonly-vanilla-nosu

arm64-ab-gapps-su

Click to collapse

Note: check patches when you use these auto scripts, if some patch is broken you’ll have build errors

8. Building using the manual way

In simple steps:

Repo init the rom you want to build GSI for.

1mkdir ~/rom && cd ~/rom
2repo init -u https://github.com/LineageOS/android.git -b lineage-15.1

Add phh repos to your local_manifest

1git clone https://github.com/phhusson/treble_manifest .repo/local_manifests -b android-8.1

After git clone you need to remove replace.xml if you’re building any rom expect aosp.

phhusson said:

The replace.xml is meant only for AOSP, if you build a custom ROM, remove it, but don’t forget to cherry-pick the patches.

Sync the source

1repo sync -c -j4 --force-sync --no-tags --no-clone-bundle

Modify the source to fix issues in other devices using one of these methods:

Apply phh patches:

1git clone https://github.com/phhusson/treble_patches -b android-8.1

Then apply each path in its project

1patch -p1 < patch

Or cherry-pick changes by phhusson from here, remember to choose the latest branch platform_build - platform_external_selinux - platform_frameworks_av - platform_frameworks_base - platform_frameworks_native - platform_frameworks_opt_telephony - platform_system_bt - platform_system_core - platform_system_libvintf - platform_system_vold

1git fetch repo branch && git cherry-pick commit

Go to the phh device repo and edit the .mk for your ROM (example lineage.mk)
Lunch the build varaint you want (ex. treble_arm64_avN-userdebug) and start the build

1. build/envsetup.sh
2lunch treble_arm64_avN-userdebug
3WITHOUT_CHECK_API=true make -j8 systemimage

If you want to compress the system image after build finishes, go to out/target/product/phh_*/ folder and run

1xz -c system.img > system.img.xz

Credits:

@phhusson for all his contributions, without his efforts this can’t be possible.
@Dakkar for his build script
@fAIyaZ for helping me build my first GSI
@sooti for his simplified instruction on phh-treble telegram.
@lineageos guys for their wiki

Note: This article was posted originally on XDA Developers forums, you can find it here.

DualBootPatcher Ultimate Guide: How to use, build and add new devices!

Sat, 26 Aug 2017 00:00:00 +0000

Introduction & How to use DualBootPatcher?

What is DualBootPatcher?

DualBootPatcher is an open-source app that allows multiple ROMs to be installed on a single Android device. It does its best to work with existing code and does not require explicit support from ROMs. There are currently 270+ supported devices and their variations. It’s originally developed by the amazing developer @chenxiaolong with help of many contributors

What does this app do ?

It patches…

Custom kernels for dual boot support
ROMs so that they can be installed as secondary
Google Apps packages for AOSP-based ROMs
SuperSU so that it can be used in the secondary ROM

Where can i find it?

What’s supported ?

Except Toaster and Alarm clocks pretty much everything is supported.

How to use the App?

Download, install and open the app.
Swipe to the right to open the menu. Click “ROMS”. Now if this is the first time you use it, it will ask you if you want to set kernel. Do so!
After it has finished go to ROM Settings (primary ROM 3 dot menu) and select Update Ramdisk. It will update it and will ask you to reboot. Press Reboot Now, or Reboot later.
Now Download any ROM you like and open the app again and open the menu and open Patch Zip File from the menu. Ensure that your Device is set to (yourdevicename) and under Partition configuration select secondary/dataslot (will install 2nd ROM in /system) or data slot.
Click continue and select where to save the patched file.
You should see the file is being put in “Queue”. Just click the confirm button to the upper right. Note: If you want to go back, just swipe the ROM in queue to right and start over.
The app will patch the zip. When done, go back to “ROMs”.
Click “Flash zip files” (the big pink button on the lower right). Click the pink plus button to add your previously patched zip file.
Locate the file you have patched in step 7. Unless you have changed the name there, it should be something like ROM_name_partition_config_ID.zip (like RR-N-v5.8.3-20170707-cheeseburger-Unofficial_dual.zip).
Click on that file and choose “Keep location”. Now confirm the flash with the button on the upper right side. Note: You can also install the patched zip files in recovery.
It will now open the terminal and begin flashing the file. This requires some patience. After it has flashed the file you’ll see success message in green.
Now click back and you should see your newly installed ROM along with the Primary ROM. Note: You can find more options by clicking on the three buttons on each ROM.
Now reboot and wait till finishing 2nd ROM first boot. install DualBootPatcher apk so you can easily switch ROMs, there is another way to change ROMs: flash DualBootUtilities.zip and switch ROM manually.

Using Bootui:

Open app then select settings and press install (update) bootui. then Swipe to the right to open the menu. Click “ROMS” again and open secondary ROM Settings) and select Update Ramdisk, Now you can change ROMs simply using boot ui (something like grub bootloader but it works like twrp).

Partitions Configurations:

The patcher offers several locations for installing ROMs:

Primary: This is normally used for installing a zip to the primary ROM. It is not required, but is strongly recommended because it has code to prevent the zip from inadvertently affecting other ROMs.
Dual: Dual/Secondary is the first multiboot installation location. It installs to the system partition. This is a good spot for installing a second ROM because it doesn’t take any space away from the internal storage.
Multi-slots: There are 3 multislots: multi-slot-1, multi-slot-2, multi-slot-3. These install to the cache partition. This is specifically for devices, like the Galaxy S4, that have a massive cache partition.
Data-slots: There can be an unlimited number of data slots. These install to the data partition and eat up space on the internal storage. This is useful for devices where the system partition is nearly full and the cache partition is tiny. These slots are named “data-slot-[id]”, where “id” is something you provide in the app.
Extsd-slots: There can be an unlimited number of extsd slots. These install to the external SD card, which is useful as it keeps the ROMs off of the internal storage. Note that the ROM’s data files are still stored on the data partition.

DualBootPatcher very recently got support for sharing apps and their data across ROMs. Maybe sharing is somewhat of a misleading term. The feature actually makes Android load the shared apps and data from a centralized location, /data/multiboot/_appsharing. So you’re not sharing apps from one ROM to another per se. The ROMs are just loading the apps from one shared location. Let me make this clearer with an analogy.

Think of the people in a company office as ROMs. You want to share with your coworkers some documents (apps). Instead of telling them to come over to your desk to see those documents (sharing apps from one ROM to another), everyone goes to the conference room to look at the documents together (loading apps from a shared location). That’s how app and data sharing is implemented.

To use app sharing, follow these steps in every ROM that you want to use app sharing: (doesn’t work with JB ROMs)

Install the app you want to share
Open DualBootPatcher and go to “App Sharing” in the navigation drawer
Enable individual app sharing
Tap “Manage shared applications” and enable APK/data sharing for the app
Reboot

When you uninstall an app that’s shared, it simply become unshared for the current ROM. That way, other ROMs are not affected. To continue the analogy above, if you quit your job, you won’t shred the documents that everybody else was looking at.

If you unshare an app’s data, it will go back to using the data it had before it was shared. In other words, you leave the conference room and go back to work on your own documents at your desk.

FAQ:

How to boot to another ROM ?

This is simple … There is no reboot to primary, secondary or whatever. So all you have to do is:

Go to ROMs section of the App.
Click on the ROM you want to boot to. You should see “Switching ROM” message. After few seconds, you should see a report message saying that “ROM successfully switched”.
Now just do a normal reboot of your device. See the magic! It should boot to the ROM you have switched on step 2. Note: You can find more options by selecting the three buttons on each ROMs (like creating reboot widgets for directly rebooting to specific rom). You also need to install the App to all of the ROMs you install. Otherwise, you want be able to boot to other ROMs!

Wipe /cache, /data, /system, or dalvik-cache? The easiest way is to do it from the app while booted in another ROM. Just go to “Roms” in the navigation drawer, tap the 3 dots options menu for the ROM you want to wipe, and tap “Wipe ROM”. Update the primary ROM? Patch the zip for primary and flash it. The “primary” installation target is designed so that other ROMs won’t be affected when you want to flash something for the primary ROM. Update a non-primary ROM? Patch and flash the zip exactly like how you did it the first time. Flash a mod or custom kernel for the primary ROM? Patch it for primary before flashing. If the zip does not wipe /cache, it is also safe to flash it directly. Flash a mod or custom kernel for a non-primary ROM? Just patch and flash it :)

How to build DualBootPatcher from source?

When you deeply exploring DualBootPatcher repository you’ll find all information and guides you need but for some people instructions isn’t clear enough, so let me explain it here - You’ll need linux, whichever distribution you use. but i’m sure building works on [Debian-Ubuntu-Fedora-Arch] and all its derivations.

A) Prerequisites

- You’ll need these packages whatever version you want to build:

Android NDK

Currently NDK r15 is required to build, you can get it here Download, Extract and rename it to “android-ndk”

1wget https://dl.google.com/android/repository/android-ndk-r15-linux-x86_64.zip && unzip android-ndk-r15-linux-x86_64.zip && mv android-ndk-r15 android-ndk

And then export android ndk:

1echo "export ANDROID_NDK=$(pwd)/android-ndk" >> ~/.bashrc
2echo "export ANDROID_NDK_HOME=$(pwd)/android-ndk" >> ~/.bashrc

cmake

DBP needs cmake 3.7.2 or higher. you need to download it here Downloading & Extract command:

1wget https://cmake.org/files/v3.7/cmake-3.7.2.tar.gz && tar xzf cmake-3.7.2.tar.gz

extract to folder and run the following commands in cmake folder

1sed -i 's|cmake_options="-DCMAKE_BOOTSTRAP=1"|cmake_options="-DCMAKE_BOOTSTRAP=1 -DCMAKE_USE_OPENSSL=ON"|' bootstrap
2./bootstrap
3make
4sudo make install

OpenSSL

I’m have made a successful build using openssl-1.1.0c, download from here and extract.

1wget https://www.openssl.org/source/openssl-1.1.0c.tar.gz && tar xzf openssl-1.1.0c.tar.gz

open openssl-1.1.0c folder and install it

1./config shared --prefix=/usr/local/ssl --openssldir=/usr/local/ssl
2make
3sudo make install
4echo 'PATH=${PATH}:/usr/local/ssl' >> ~/.bashrc

gtest

Install on Ubuntu / Debian:

1sudo apt-get install libgtest-dev

On Fedora:

1sudo dnf install gtest-devel

On Archlinux:

1sudo pacman -S gtest

yaml-cpp

Install on Ubuntu / Debian using

1sudo apt-get install libyaml-cpp-dev

On Fedora

1sudo dnf install yaml-cpp-devel

On Archlinux:

1sudo pacman -S yaml-cpp

Other General packages that must be installed: [Skip this if you have android build environment] Ubuntu / Debian:

1sudo apt-get install ccache libboost-dev libssl-dev openssl python-minimal build-essential libfontconfig1 findutils git make libprocps-dev unzip zip gcc-multilib g++-multilib lib32ncurses5-dev transifex-client gnupg mesa-common-dev libglu1-mesa-dev

Fedora:

1sudo dnf install ccache findutils gcc-c++ git make procps-ng unzip zip gnupg ncurses-compat-libs transifex-client openssl-devel

Archlinux:

1sudo pacman -S ccache boost openssl lib32-openssl findutils git make procps-ng unzip zip gnupg gcc-multilib ncurses

1sudo pacaur -S transifex-client ncurses5-compat-libs lib32-ncurseslib32-ncurses5-compat-libs

- For building the android app you need:

Android SDK

You need to install SDK using android studio (build-tools 25.0.3, platforms android-25, platform-tools and tools) or install from terminal:

1mkdir -p android-sdk && cd android-sdk
2wget -q https://dl.google.com/android/repository/platform-tools_r25.0.3-linux.zip && unzip -qq platform-tools_r25.0.3-linux.zip
3wget -q https://dl.google.com/android/repository/build-tools_r25.0.3-linux.zip && unzip -qq build-tools_r25.0.3-linux.zip
4wget -q https://dl.google.com/android/repository/tools_r25.2.3-linux.zip && unzip -qq tools_r25.2.3-linux.zip

Then Export it:

1echo "export ANDROID_HOME=$(pwd)" >> ~/.bashrc
2echo "export PATH=${PATH}:${ANDROID_HOME}/tools" >> ~/.bashrc

JDK 1.8

Ubuntu / Debian:

1sudo apt-get install openjdk-8-jdk openjdk-8-jdk-headless

Fedora:

1sudo dnf install java-1.8.0-openjdk-devel java-1.8.0-openjdk-headless

Archlinux:

1sudo pacman -S jdk8-openjdk

Other packages: Ubuntu / Debian:

1sudo apt-get install lib32stdc++-6-dev

Fedora:

1sudo dnf install glibc.i686 libstdc++.i686

Archlinux:

1sudo pacman -S glibc libstdc++5 lib32-libstdc++5

And for linux version you need:

it must be 5.3 or higher Ubuntu / Debian:

1sudo apt-get install qttools5-dev-tools libqt5core5a

Fedora:

1sudo dnf install qt5-qtbase-devel

Archlinux:

1sudo pacman -S qt5-base

Other packages: Ubuntu / Debian:

1sudo apt-get install libarchive-dev liblz4-tool liblzma-dev lz4-dev zlib1g-dev lib32z-dev

Fedora:

1sudo dnf install libarchive-devel lz4-devel xz-devel

Archlinux:

1sudo pacman -S libarchive lz4 xz lib32-xz lzip

- Some final touches: Enable cache:

1echo "export USE_CCACHE=1" >> ~/.bashrc

Reload bash environment:

1source ~/.bashrc

- Cloning the source:

1git clone --recursive https://github.com/chenxiaolong/DualBootPatcher.git

B) Building Android App

Open DualBootPatcher folder and make new folder “build” and open it.

1mkdir build && cd build/

Now Build the app:

1cmake .. -DMBP_BUILD_TARGET=android -DMBP_BUILD_TYPE=debug
2make
3cpack -G TXZ
4make apk

And you’ll find built apk in DualBootPatcher/Android_GUI/build/outputs/apk/Android_GUI-debug.apk

C) Building Utilities Zip

Utilities Zip is AROMA based zip which enable you to switch ROMs form TWRP or wipe ROM when something went wrong. - In build folder run these commands:

1make android-system_armeabi-v7a
2make -C data/devices
3./utilities/create.sh

You’ll find built utilities.zip in DualBootPatcher/build/utilities/DualBootUtilities-9.2.0.zip

D) Building Linux App

In build folder run these commands:

1cmake .. -DMBP_BUILD_TARGET=desktop -DMBP_PORTABLE=ON
2make
3cpack -G TXZ

or to pack it to zip file:

1cpack -G ZIP

How to build DualBootPatcher using docker images?

What is docker? [in case you don’t know it [😕]

Docker is the world’s leading software container platform. Developers use Docker to eliminate “works on my machine” problems when collaborating on code with co-workers. Operators use Docker to run and manage apps side-by-side in isolated containers to get better compute density. Enterprises use Docker to build agile software delivery pipelines to ship new features faster, more securely and with confidence for both Linux, Windows Server, and Linux-on-mainframe apps.

Firstly, make sure you have docker installed, if not install it From here you should choose a method you’ll follow, Building docker images manually or use pre-built docker images.

A) Building docker images manually:

@chenxiaolong has made docker image configuration files here all you need to do after sync the repo like part 1 is entering the following commands in DualBootPatcher directory:

1./docker/generate.sh
2./docker/build.sh

Note that building the docker images will take a long time and consume a lot of bandwidth–multiple gigabytes at the very least. It will download all the dependencies for building DualBootPatcher for all supported targets.

B) Using ready-made docker images:

I have made docker images and uploaded them to docker hub to download “pull” it enter these commands:

1docker pull yshalsager/dualbootpatcher:9.3.0-4-base
2docker pull yshalsager/dualbootpatcher:9.3.0-4-android
3docker pull yshalsager/dualbootpatcher:9.3.0-4-linux

C) Building DualBootPatcher using docker:

- To enter docker container and build DualBootPatcher make folder “builder” in DualBootPatcher directory.

1cd DualBootPatcher && mkdir -p builder

- Then run this to enter android build container

1docker run --rm -it -e USER_ID=$(id -u) -e GROUP_ID=$(id -g) -v "$(pwd):/builder/DualBootPatcher:rw,z" -v "${HOME}/.android:/builder/.android:rw,z" -v "${HOME}/.ccache:/builder/.ccache:rw,z" -v "${HOME}/.gradle:/builder/.gradle:rw,z" yshalsager/dualbootpatcher:9.3.0-4-android bash

- After this you’ll continue like normal building:

1cd DualBootPatcher/builder && cmake .. -DMBP_BUILD_TARGET=android -DMBP_BUILD_TYPE=debug && make -j16 && rm -rf assets && cpack && make apk -j16
2make android-system_armeabi-v7a -j16 && make -C data/devices -j16

- Now exit from container and enter linux build one:

1docker run --rm -it -e USER_ID=$(id -u) -e GROUP_ID=$(id -g) -v "$(pwd):/builder/DualBootPatcher:rw,z" -v "${HOME}/.android:/builder/.android:rw,z" -v "${HOME}/.ccache:/builder/.ccache:rw,z" -v "${HOME}/.gradle:/builder/.gradle:rw,z" yshalsager/dualbootpatcher:9.3.0-4-linux bash

- Build utilities ZIP:

1cd DualBootPatcher/builder && ./utilities/create.sh

- And build linux app

1cmake .. -DMBP_BUILD_TARGET=desktop -DMBP_PORTABLE=ON && make -j16 && cpack

- You’ll find output files in your local “builder” folder like normal build.

How to build DualBootPatcher using TravisCI?

Travis CI is free open-source continues integration service, which simply take your project from GitHub and test / build it for you ! it’s amazing solution if you don’t have time/free space/mind lol :cyclops: to test your code. - you’ll need to signup Travis account, go here and press signup, enter your GitHub account details then approve Travis needed permissions and done. - Now Fork DualBootPatcher repository to your account and create .travis.yml file in project root. - Copy my Travis configuration to your .travis.yml

 1sudo: required
 2
 3services:
 4 - docker
 5
 6before_install:
 7 # Clone DualBootPatcher Repository
 8 - git clone --recursive https://github.com/yshalsager/DualBootPatcher -b master DualBootPatcher/
 9 # Pull docker images
10 - docker pull yshalsager/dualbootpatcher:9.3.0-4-base
11 - docker pull yshalsager/dualbootpatcher:9.3.0-4-android
12 - docker pull yshalsager/dualbootpatcher:9.3.0-4-linux
13script:
14 # Make work directories 
15 - mkdir $HOME/.android
16 - mkdir -p ${TRAVIS_BUILD_DIR}/DualBootPatcher/builder/ && cd ${TRAVIS_BUILD_DIR}/DualBootPatcher/
17 # Build APK
18 - |
19 docker run --rm -i -e USER_ID=$(id -u) -e GROUP_ID=$(id -g) -v "$(pwd):/builder/DualBootPatcher:rw,z" -v "${HOME}/.android:/builder/.android:rw,z" yshalsager/dualbootpatcher:9.3.0-4-android bash << EOF
20 cd DualBootPatcher/builder && cmake .. -DMBP_BUILD_TARGET=android -DMBP_BUILD_TYPE=debug && make -j16 && rm -rf assets && cpack && make apk -j16
21 make android-system_armeabi-v7a -j16 && make -C data/devices -j16
22 exit
23 EOF
24 - |
25 docker run --rm -i -e USER_ID=$(id -u) -e GROUP_ID=$(id -g) -v "$(pwd):/builder/DualBootPatcher:rw,z" -v "${HOME}/.android:/builder/.android:rw,z" yshalsager/dualbootpatcher:9.3.0-4-linux bash << EOF
26 # Build Utilities Zip
27 cd ~/DualBootPatcher/builder && ./utilities/create.sh
28 # Build Linux
29 cmake .. -DMBP_BUILD_TARGET=desktop -DMBP_PORTABLE=ON && make -j16 && cpack
30 exit
31 EOF
32after_success:
33 - export TRAVIS_CURRENT_DATE=$(date +"%d%m%y-%Hh%Mm")
34 # Check output & md5sum
35 - ls -l ${TRAVIS_BUILD_DIR}/DualBootPatcher/Android_GUI/build/outputs/apk/debug/Android_GUI-debug.apk
36 - md5sum ${TRAVIS_BUILD_DIR}/DualBootPatcher/Android_GUI/build/outputs/apk/debug/Android_GUI-debug.apk
37 - ls -l ${TRAVIS_BUILD_DIR}/DualBootPatcher/builder/utilities/DualBootUtilities-9.3.0.zip
38 - md5sum ${TRAVIS_BUILD_DIR}/DualBootPatcher/builder/utilities/DualBootUtilities-9.3.0.zip
39 - ls -l ${TRAVIS_BUILD_DIR}/DualBootPatcher/builder/DualBootPatcher-9.3.0-Linux.zip
40 - md5sum ${TRAVIS_BUILD_DIR}/DualBootPatcher/builder/DualBootPatcher-9.3.0-Linux.zip
41 # Upload to transfer.sh
42 - cd ${TRAVIS_BUILD_DIR}/DualBootPatcher/Android_GUI/build/outputs/apk/debug/ && curl --upload-file ./Android_GUI-debug.apk https://transfer.sh/Android_GUI-debug-${TRAVIS_CURRENT_DATE}.apk
43 - cd ${TRAVIS_BUILD_DIR}//DualBootPatcher/builder/utilities/ && curl --upload-file ./DualBootUtilities-9.3.0.zip https://transfer.sh/DualBootUtilities-9.3.0-${TRAVIS_CURRENT_DATE}.zip
44 - cd ${TRAVIS_BUILD_DIR}/DualBootPatcher/builder/ && curl --upload-file ./DualBootPatcher-9.3.0-Linux.zip https://transfer.sh/DualBootPatcher-9.3.0-${TRAVIS_CURRENT_DATE}-Linux.zip

- Edit line 8 with your Repository and branch you want to build.

1 - git clone --recursive https://github.com/yshalsager/DualBootPatcher -b master DualBootPatcher/

- This simple TravisCI configuration builds both of apk and utilities.zip and deploy them to transfer.sh. - Now open Travis and add your DualBootPatcher fork to projects and start building. - You’ll find built apk md5 and links from line at the end Travis log - Build takes about 25 mins. - When you add new commits to repository Travis will always build new apk for you! - If you want to change something in configuration, always check its syntax to make sure it’s valid YAML using this site.

How to add new Devices to DualBootPatcher?

To add your/new device to DualBootPatcher you need to get some information first then add it to DualBootPatcher devices: - Flash GetLogs-20161128-1.zip using recovery [TWRP highly recommended] - Copy /sdcard/logs/[Date&Time].tar to PC, extract it and let’s start. - This is new device template:

 1- name: (Device Name)
 2 id: (device id)
 3 codenames:
 4 - (device codename1)
 5 - (device codename2)
 6 - (device codename3)
 7 architecture: (device architecture)
 8
 9 block_devs:
10 base_dirs:
11 - (/dev/block/bootdevice/by-name)
12 system:
13 - (/dev/block/bootdevice/by-name/system)
14 cache:
15 - (/dev/block/bootdevice/by-name/cache)
16 data:
17 - (/dev/block/bootdevice/by-name/userdata)
18 boot:
19 - (/dev/block/bootdevice/by-name/boot)
20 recovery:
21 - (/dev/block/bootdevice/recovery)
22 extra:
23 - (/dev/block/bootdevice/modem)
24
25 boot_ui:
26 supported: true
27 flags:
28 - (TW_HAS_DOWNLOAD_MODE)
29 graphics_backends:
30 - fbdev
31 brightness_path: (/sys/class/leds/lcd-backlight/brightness)
32 max_brightness: (255)
33 default_brightness: (162)
34 pixel_format: (RGBX_8888)
35 theme: portrait_hdpi

- You’ll edit all values between brackets. - open logs/system/build.prop, you’ll find: ro.product.model= is your (Device Name) ro.product.name= and ro.product.device= is your (device id) and (device codename1), if you know more names for your device add it in (device codename2) / (device codename3) ro.product.cpu.abi= is your (device architecture) - open logs/recovery/recovery.fstab, you’ll find main device mount points, copy each mount point to it name - open logs/recovery/recovery.log, you’ll find TW_BRIGHTNESS_PATH := /sys/class/leds/lcd-backlight/brightness this is your (brightness_path) I:Got max brightness 255 is (max_brightness) setting GGL_PIXEL_FORMAT_RGBA_8888 is (pixel_format) - Note: you can get these info from BoradConfig.mk in your device twrp tree. - Now read “Partition Logs:” in recovery.log it’ll tell us second mount point, as example /system | /dev/block/mmcblk0p41 so you should add it like

1 system:
2 - /dev/block/bootdevice/by-name/system
3 - /dev/block/mmcblk0p41

and so on with other partitions. - Checking more mounts: (not necessary but i recommend it) open logs/listings/dev_full and scroll down to /dev/block/platform/soc.0 you’ll find another main mount point which contain by-name and by-num folders like /dev/block/platform/soc.0/f9824900.sdhci from /dev/block/platform/soc.0/f9824900.sdhci/by-name list copy each partition mount point to your device - Now we’re done, you should have something like

 1- name: Xiaomi Mi4S
 2 id: aqua
 3 codenames:
 4 - aqua
 5 architecture: arm64-v8a
 6
 7 block_devs:
 8 base_dirs:
 9 - /dev/block/bootdevice/by-name
10 - /dev/block/platform/soc.0/f9824900.sdhci/by-name
11 system:
12 - /dev/block/bootdevice/by-name/system
13 - /dev/block/platform/soc.0/f9824900.sdhci/by-name/system
14 - /dev/block/mmcblk0p41
15 cache:
16 - /dev/block/bootdevice/by-name/cache
17 - /dev/block/platform/soc.0/f9824900.sdhci/by-name/cache
18 - /dev/block/mmcblk0p42
19 data:
20 - /dev/block/bootdevice/by-name/userdata
21 - /dev/block/platform/soc.0/f9824900.sdhci/by-name/userdata
22 - /dev/block/mmcblk0p44
23 boot:
24 - /dev/block/bootdevice/by-name/boot
25 - /dev/block/platform/soc.0/f9824900.sdhci/by-name/boot
26 - /dev/block/mmcblk0p37
27 recovery:
28 - /dev/block/bootdevice/by-name/recovery
29 - /dev/block/platform/soc.0/f9824900.sdhci/by-name/recovery
30 - /dev/block/mmcblk0p38
31
32 boot_ui:
33 supported: true
34 graphics_backends:
35 - fbdev
36 flags:
37 - TW_QCOM_RTC_FIX
38 pixel_format: RGBX_8888
39 brightness_path: /sys/class/leds/lcd-backlight/brightness
40 max_brightness: 255
41 default_brightness: 162
42 theme: portrait_hdpi

- Open DualBootPatcher/data/devices/(your device manufacturer).yml and add your device codes. then Check your code syntax here if everything is right, make commit to add your device to your repository. - Build apk using any method explained above to test it.

How to submit new Devices to official DualBootPatcher support?

Once you checked everything is working, it’s time to contribute to DualBootPatcher and add your device to official devices: - First, Check you made all needed changes to add your device without problems, again check for YAML syntax because if it’s wrong build won’t work. - Open DualBootPatcher Repository, press New pull request button. - Press compare across forks and change the head fork to your username/DualBootPatcher - Give a name and a description to your pull request, some thing like: “Add (Device Name) support” or “data: Add (device name) - Now click the “Create pull request” green button - Wait for your changes to be verified and merged to the project, the developer received your code and he will review, try and add it to the source code. You will be notified when your code merged.

Note: This article was posted originally on XDA Developers forums, you can find it here.

How to use more than keyboard layout and switch between them in Android x86?

Sat, 18 Jun 2016 00:00:00 +0000

Android x86 supports multiple keyboard layouts but it doesn’t support switching between them easily using a shortcut like alt+shift. Here’s how you can do this!

1- Install External Keyboard Helper Pro App.

2- Open Setting - Language & input, then active External Keyboard.

3- Open External Keyboard Helper Settings, Choose your first keyboard layout then open Advanced settings.

4- Choose Language switching.

5- Open Language-switch key.

6- Check Alt button, Press change then press ALT button from keyboard, after that press Save.

7- Choose your second keyboard layout.

Note: This article was posted originally on XDA Developers forums, you can find it here.

Posts on yshalsager

Using Parsel instead of Beautiful Soup for Web Scraping

Thoughts and Tricks After Usage

Extra Resources

Debugging and fixing Selenium's send_keys() wrong text input

How to build a Project Treble GSI ROM from source?

1. Installing SDK

2. Installing build packages

3. Installing the repo command

4. Configuring git

5. Turning on caching to speed up build

6. Building using phhusson’s script

7. Building using dakkar’s script

8. Building using the manual way

DualBootPatcher Ultimate Guide: How to use, build and add new devices!

Introduction & How to use DualBootPatcher?

What is DualBootPatcher?

What does this app do ?

Where can i find it?

What’s supported ?

How to use the App?

Using Bootui:

Partitions Configurations:

Apps and Data sharing:

FAQ:

How to build DualBootPatcher from source?

A) Prerequisites

B) Building Android App

C) Building Utilities Zip

**D) Building Linux **App

How to build DualBootPatcher using docker images?

A) Building docker images manually:

B) Using ready-made docker images:

C) Building DualBootPatcher using docker:

How to build DualBootPatcher using TravisCI?

How to add new Devices to DualBootPatcher?

How to submit new Devices to official DualBootPatcher support?

How to use more than keyboard layout and switch between them in Android x86?

D) Building Linux App