How do I create a recipe for content with these attributes

[anleva]

Hello -

I've been messing around with recipes but haven't yet been able to get a clean result. For my news site content I notice that unwanted content uses a particular article tag and wanted another. For example:

<article class="asset clearfix">
...
...
...
</article>


<article class="asset story clearfix">
...
...
...
</article>

The former class includes photo galleries and video stories which I don't want any content from. The latter is news stories.

What do i need to do in my recipes to ignore all pages with the former content and only include the latter "asset story clearfix"?

Thanks

[kovidgoyal]

keep_only_tags = dict(attrs={'class':'asset story clearfix'})

def preprocess_raw_html(self, html, url):
    if '<article class="asset clearfix">' in html:
         self.abort_article()
    return html

[anleva]

Originally Posted by kovidgoyal

keep_only_tags = dict(attrs={'class':'asset story clearfix'})

def preprocess_raw_html(self, html, url):
    if '<article class="asset clearfix">' in html:
         self.abort_article()
    return html

Thank you I will try this. If I also add the remove_tag with additional attributes does it matter where it is placed in the recipe? Same with auto_cleanup?

[kovidgoyal]

You cannot use auto_cleanup and keep/remove together. It is one or the other.

[anleva]

Thank you. I am also a bit confused about the syntax.

In your example above you used:

keep_only_tags = dict(attrs={'class':'asset story clearfix'})

I was thinking it would be something link

keep_only_tags = dict(name='article', attrs={'class':'asset story clearfix'})

as for remove tags I sometimes see:

remove_tags = [dict(name='div', attrs={'class':'advert'})]

When do you need to have name='xx' in the string and when do you not need it?

[kovidgoyal]

You dont ever need it unless you are trying to distinguish between two types of tags that have the same value for class.

[anleva]

Thank you for your help! It looks nearly perfect now.

One final thing to try and clean up. As desired, it is not parsing any content from the article class that I do not want, i.e. <article class="asset clearfix">. However those article titles are in the table of contents. When i click on them it does not go to any content but to the next story with content. How can I eliminate their article titles from appearing in the TOC?

Amazing software!

[anleva]

Originally Posted by anleva

Thank you for your help! It looks nearly perfect now.

One final thing to try and clean up. As desired, it is not parsing any content from the article class that I do not want, i.e. <article class="asset clearfix">. However those article titles are in the table of contents. When i click on them it does not go to any content but to the next story with content. How can I eliminate their article titles from appearing in the TOC?

Amazing software!

Any ideas?