Selenium Erorr: Failed to establish a new connection

I was trying to auto access to a site (stackoverfrom.com). While doing try and error with selenium python, I got this error:

MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=48710): Max retries exceeded with url: /session/2a452d24-4918-4c3c-858d-22eb739d604c/element (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))

I thought my IP address was blocked (by stackoverflow.com)

But it rurned out that this was coming from different part.

Driver.quit was done because wait.WebDriverWait(driver,30).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "XXXXX"))) was wrong.

That is: Class name isn't available for WebDriverWait.


https://stackoverflow.com/questions/57262217/how-do-you-use-ec-presence-of-element-locatedby-id-mydynamicelement-excep

Selenium find element by class FOR MULTIPLE CLASSES

When an element of web page has multiple classes, just set one class capture the element.

Like:

   

Use find element by class name

content = driver.find_element_by_class_name('content')

https://selenium-python.readthedocs.io/locating-elements.html


In above case, if class name "btn-red" is unique for the page,

content = driver.find_element_by_class_name('btn-red')

work.

seleniumでMessage: newSessionというエラーが出るので直す

webdriver.Firefoxを使っていたら

WebDriverException: Message: newSession

というエラーが出て停止しました。少し放置していたのでgeckodriverがバージョンアップしているということでした。

https://stackoverflow.com/questions/46439714/selenium-common-exceptions-webdriverexception-message-session-not-created-exce


最新のgeckodriverをダウンロードして、

$ sudo mv geckodriver /usr/local/bin

で/usr/local/binに移動させると復活しました。


(ubuntu16.04, Python)

[Python] Get title and post date of Blogger

Script used for only Blogger (blogspot, like this blog) . Use Selenium and Python.
 

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
target_url = ''### Blogger only, blogspot.jp or...
num_get = 3 ### how many pages to check
result = []
try:
    b = webdriver.Firefox()
    time.sleep(3)
except:
    quit()
b.get(target_url)
wait = WebDriverWait(b, 30).until(
    EC.element_to_be_clickable((By.CLASS_NAME, 'post-title')))
newest_title = b.find_elements_by_class_name('post-title')[0].text
newest_postlink = b.find_elements_by_partial_link_text(newest_title)[0].get_attribute('href')
b.get(newest_postlink)
# get post date and title, move to older page
cnt = 0
while cnt < num_get:
    wait = WebDriverWait(b, 30).until(
        EC.element_to_be_clickable((By.CLASS_NAME, 'date-header')))
    date = b.find_elements_by_class_name('date-header')[0].text
    
    title = b.find_elements_by_class_name('post-title')[0].text
    
    print(date, title)
    result.append([date, title])
    b.find_elements_by_class_name('blog-pager-older-link')[0].click()
        
    cnt += 1
b.close()

If you want to get result with csv

import pandas as pd
pd.DataFrame(result).to_csv('date_title.csv', header=None, index=None)